Is it possible to set the logical sector size of a USB hard disk drive?

I have two supposedly identical 3TB Western Digital USB hard disk drives. One contains nearly the full 3TB of data. I want the other to be a backup of that data. There are lots of small files, so running cp would be even slower than copying 3TB over USB using dd. Both drives were bought at the same time, both use MSDOS partition table, but one shows logical/physical sector sizes of 4096/4096 and the other shows 512/4096.

My problem is that I’m trying to copy to the 512/4096 drive, but I can’t create a large enough partition because of the limits of the MSDOS partition table with 512 byte (logical) sectors. I've tried various commands (dd of the first few sectors to partition table, parted, fdisk, gparted, Windows XP “Manage” and Windows 7 “Manage”) but end up with either 2TB max limits or ~300GB partitions that thought they were 3TB when they were created.

Given that one of my drives is running at 4096/4096 then this question doesn’t hold true for my case.

Using GPT is my fall-back option, but it creates a partition that is slightly smaller than the NTFS partition, so I wouldn't be able to just dd the filesystem on the partition.

Is there a way to force 4096/4096 for an MSDOS partition table (either from Linux or Windows) so that I can create a 3TB partition that perfectly matches my source disk?

IBBoard

Posted 2012-08-20T14:20:40.363

Reputation: 221

Couldn't you use software to simply duplicate the entire disk sector by sector? I know Acronis and CloneZilla have this ability. – Ramhound – 2012-08-20T15:04:40.693

1Why can't you create a FAT32 partition with a sector size greater than 512? That's what is limiting the size of the volume. FAT32 maximum volume sizes range from 2 TB for a sector size of 512 bytes up to 16 TB for a sector size of 4,096 bytes. – martineau – 2012-08-20T15:18:25.590

@Ramhound - I've used dd if=/dev/sdb of=/dev/sdc, which is a Linux command that does sector by sector copying of an entire device (not just a partition), but the partition on the second device comes out at ~300GB despite being a perfect copy. dd is one of Clonezilla's 'last ditch' options when it can't do anything more intelligent. – IBBoard – 2012-08-20T18:27:56.940

@martineau: I'm using NTFS, because that's what the source partition was. I've mounted the drive in Win7, opened "Manage" and if it is on "MBR" partitioning (which the first disk appears to be) then even Win7 won't let me create a partition larger than 2TB. When printing the disk info in Linux using fdisk (IIRC) then "Sector size (logical/physical)" definitely differs between devices, which screws the partitions I have made. – IBBoard – 2012-08-20T18:31:06.373

Western Digital has free disk utilities for partitioning their drives -- you could try using them. Personally I use Acronis Disk Director -- a commercial program -- to do virtually all my partition manipulations. – martineau – 2012-08-20T19:28:56.653

Answers

I just had a similar issue. I had a Seagate 3TB USB Desktop Expansion drive that I wanted to move into my PC for faster access (I only had USB 2.0 on the PC). Once I did that I couldn't access the filesystem. Returning it to the USB to SATA controller made it work again. Comparing the two I found that in the external case (using the USB to SATA controller) the drive had 4096/4096 logical/physical sector sizes and when connected to the internal motherboard SATA, it had 512/4096 logical/physical sector sizes. Much like the OP sugguests, a 4096 logical sector size allows for > 2TiB partitions, but at the 512 sector size we get the 2TiB limit. I didn't want to copy stuff off (slowly over USB) to somewhere, repartition (GPT), reformat, and copy back, so I kept at it. My ultimate solution on Linux was to:

Delete the original MBR partition table which uses 4k sectors. Use gdisk (GPT variant of fdisk) to create the partition using 512 byte sectors spanning the exact same area. This basically amounted to taking the sectors for start/end of the partition in 4K-land and multiplying by 8. I did have an off-by one on the end, so I added 8 more there and so far so good (e2fsck alerted me to that.) The e2fsk is still going, so I'm not 100% sure I'm safe, but it seems all good in theory and I did mount the files for a bit and they looked fine.

Shane Anderson

Posted 2012-08-20T14:20:40.363

Reputation: 51

I can confirm this approach works. This was my partition table after a night-long rsync and after having moved the disk from a USB-to-SATA adapter to a proper SATA connector (if you do the sums, you'll notice a perfect factor of 8). I've first tried a few times with a partial dd image and then I took the plunge. BTRFS scrub was totally happy with the values. Hope I'll never have to do this again, but it has been a didactic moment.

– Avio – 2018-11-30T14:39:07.757

Converting MBR->GPT can solve the problem of accessing a >2 TB drive originally partitioned under a different sector size, but be cautioned that not all MBR drives have enough free space at the start and end for the GPT data structures: https://www.rodsbooks.com/gdisk/mbr2gpt.html Using gdisk on the block device directly in this scenario isn't safe because gdisk will have incorrect info about where the original partition data was due to the sector size mismatch & it could clobber the partition data. Maybe use losetup --sector-size to emulate the original sector size and gdisk the loop device.

– balazer – 2019-12-27T08:15:36.903

So Linux tools seem to work okay and ignore the 2TiB limit? Interesting, but unfortunately I needed the HDD for Windows, so it had to be NTFS. Even if Ext3 lets you go past 2TiB, I suspect Windows will get upset or ignore it. – IBBoard – 2013-01-28T20:03:41.320

1It's a GPT partition table versus a MSDOS partition table. No one mentioned filesystems. You can do a conversion from MSDOS/MBR to GPT using gdisk relatively safely. – justbrowsing – 2013-09-07T22:25:04.003

There are many other reasons to want to move to a GPT, even if you have a disk smaller than 2TiB, safety and redundancy chief among them. You may wish to read this excellent article, http://www.ibm.com/developerworks/linux/library/l-gpt/index.html

– MariusMatutiae – 2013-11-07T07:35:21.410

1the limit does not related to Linux or Windows. It's just because MBR doesn't support drives larger than 2TB – phuclv – 2014-01-10T10:57:38.413

Many Western Digital external USB drives over 2 TB (including at least some My Book, My Book Essential, Elements, and Easystore drives) can be configured for either 512 byte logical sectors or 4096 byte logical sectors using the WD Quick Formatter tool. When configured for 4096 byte logical sectors, the USB-to-SATA bridge in the enclosure does a translation between 512 byte logical sectors at the SATA interface to the internal drive and 4096 byte logical sectors at the USB interface to the host computer.

With 512 byte logical sectors, an MBR partition table could only use up to 2 TB of a drive. That's because MBR table entries are 32 bits with a max of 2^32 or 4,294,967,296 sectors. 2^32 sectors x 512 bytes/sector is 2 TB. With 4096 byte logical sectors, an MBR partition cable can use up to 16 TB of a drive. (2^32 sectors x 4096 bytes/sector is 16 TB) Windows XP only supports MBR partition tables, and so 4096 byte logical sectors are the only way to use all of the space on a drive over 2 TB in Windows XP. The newer GPT partition table format supported in Windows Vista and later does not have the 32-bit limitation, and can support disks larger than 2 TB regardless of the sector size.

The WD Quick Formatter tool (version 2.0.0.65 available for download as of this writing) can enable or disable the sector size translation. This version of the tool does not work correctly under Windows XP, so I recommend running the tool in Windows 7 or later. When run in Windows 7, or in later version of Windows but with Windows 7 compatibility mode, the tool will present two configuration options on the "Format your WD external drive" screen:

XP Compatible: This option configures the drive for 4096 byte logical sectors, and creates an MBR partition table.
Most Compatible (Vista or later required): This option configures the drive for 512 byte logical sectors, and creates a GPT partition table.

Both options also create a single partition filling the entire drive, and quick format it in NTFS.

If you run the tool in Windows 8 or later without putting it in Windows 7 compatibility mode, the tool will not present a compatibility option on the formatting screen and it will format in the "most compatible" mode (512 byte logical sectors).

WD Quick Formatter 2.0.0.65 doesn't work correctly in Windows XP: it successfully configures 4096 byte logical sectors, but fails to format correctly. Another tool can subsequently be used to partition and format the drive.

WD Quick Formatter 1.2.0.10 works correctly in Windows XP, but is not available for download from Western Digital anymore.

When the target drive is configured for the same logical sector size as the source, you can copy to it using dd and the copy will work without any need to alter the partition table.

I can confirm that these Western Digital external USB drives can be configured for XP compatibility:

Easystore 12 TB (WDBCKA0120HBK)
Easystore 8 TB (WDBCKA0080HBK)
My Book Essential 1140, 3 TB (WDBACW0030HBK)
Elements 1021, 3 TB (WDBAAU0030HBK)

balazer

Posted 2012-08-20T14:20:40.363

Reputation: 86

This is a great and highly overrated answer. I was especially curious about the numbers, in particular how 2^32 translates to a 2TB limit. – Hashim – 2019-12-25T15:59:33.460

Your issue is likely similar to How to correct 512-byte sector MBR on a 4096-byte sector disk?. In short, it's the limitation of the USB enclosure's firmware by not understanding the 4K sector format

The physical sector size is the HDD's native sector size, so there's no way to "force" it to change that value unless you find someway to update the drive's firmware and do a low-level format

Some (old) 4K HDD may support emulated 512-byte sector mode (called 512e, which is what your 512/4096 drive does) in which physical size is still 4K, but it reports as having a 512-byte sector which is the logical size. That means partitions must be aligned properly to 4K or performance will be affected greatly. Technically that reported logical value may be changed by some disk-configuring software or by updating firmware if the manufacturer supports.

However I'm not aware of any such software. If you see different logical values for the same disk then it's because the old disk controller in the external enclosure doesn't understand the new 512e or 4K format. That happens a lot to USB HDDs, like mentioned in Shane Anderson's answer and:

In fact many old USB enclosures don't even support drives larger than 2TB

Worse yet, some enclosures do the reverse thing by combining 512-byte sectors to a 4K one

Complicating matters is certain USB disk enclosures. Some of these enclosures do the reverse of what AF does: They take eight disk sectors and bundle them into one new 4096-byte sector. I'm not sure what the reasoning is behind this move, but one practical advantage is that disks larger than 2TiB can be used with the old MBR partitioning system. One major disadvantage is that a disk partitioned in one of these enclosures can not be used directly or in an enclosure that doesn't do this type of translation. Likewise, a disk prepared without this translation can't be used when it's transferred into such an enclosure.

How to correct 512-byte sector MBR on a 4096-byte sector disk?

Similar issues:

Besides, you shouldn't use dd for copying disks. Instead, use some disk cloning tools, they'll smart enough to skip blank clusters and are also capable of cloning between disks with different sizes

Also, there's no "DOS partition table" but MBR and GPT (and some other uncommon formats). Unfortunately gparted and many Linux tools call that "msdos" which is incorrect

With MBR you can only addressed 2³² sectors (2TB with 512-byte sector and 16TB with 4K sector). That's why your 4096/4096 drive works properly. I suppose Shane Anderson's workaround works, but if it doesn't the only way is to use GPT. In fact GPT would be recommended even if you have only one partition because it supports checksumming and a backup partition table, which significantly reduces the chance of accidental corruption/deletion

phuclv

Posted 2012-08-20T14:20:40.363

Reputation: 14 930

it doesn't change. It's the USB disk controller that reports a different value. Disk cloning tools may not be much more efficient, but they can handle the difference in partition sizes gracefully. See my edit – phuclv – 2019-09-08T02:05:12.873

Thanks for the edits, @phuclv. The link may be relevant and misaligned partitions could have caused the change in reported sector sizes, but I don't think the other edits would have solved my specific problem (the disks are long gone, so I can't check). My issue was two supposedly identical disks, both on USB, not one disk in two caddies (where you might expect firmware to change). – IBBoard – 2019-09-08T09:20:26.403

1If there's no way to force it, how did it change? That is the crux of the problem. An identical disk (make, model, etc) shows 4K sectors. The 512b sector disk used to report the correct size, but now reports a much smaller size. /Something/ changed, so unless it is a hardware failure of some intermediary level that faked the size then it should be possible to change it back. As for the "msdos" partition table - tell that to GParted, because that's what they give you as an option. I used dd because the disk was 95% full, so ghosting apps won't be much more efficient. – IBBoard – 2014-01-18T12:02:19.667

Yes and no.

You cannot change the physical sector size, and few drives will let you change the transport layer sector size, so you will always be writing a multiple of 4K at a time.

However, Logical sector size has nothing to do with physical sector size, and the Operating system can handle such conversions by combining writes.

The drive will ALWAYS be accessed 4096 bytes at a time. That said, any modern operating system's block device driver can convert between sector sizes, better known these days as Block Sizes, and thus partition block sizes don't have to match the block size of the underlying device.

You can easily use 4K logical sector partitions on a 512 byte logical sector device, or a 512 byte logical sector partition on a 4K logical sector device.

Additionally, partition level logical sector does not have to match the sector block size used in the filesystem.

Linux and by extension Android, uses Block size (which is filesystem level logical sector sizes) of 1024, 2048 or 4096 on ExtFS. (On 512 byte sector drives it will use multiple physical sectors to form a logical block).

On FAT32, DRDOS 6.x supported larger disks by using 8K logical Sectors, and Linux can use still larger logical sectors. This allows much larger FAT32 partitions to be created.

Similarly, many software programs when managing very small filesystems that contain a large number incredibly small files have used programs with logical sector sizes as small as 64KB.

References:

Linux MSDOSTOOLS man pages

Linux mke2fs man page

Wikipedia pages on FAT:

https://en.wikipedia.org/wiki/File_Allocation_Table

https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system

Robert Wm Ruedisueli

Posted 2012-08-20T14:20:40.363

Reputation: 167

This answer is mostly wrong and would seem to confuse a drive's logical sector sizes with a filesystem's allocation unit size (a.k.a. cluster size or block size). The question is asking about a drive's logical sector size, and not about a filesystem's block size. A drive's logical sector size is not under the control of the operating system. – balazer – 2019-12-26T22:22:45.113

No, BLOCK conversion handling is done on the DEVICE DRIVER layer, not the FILESYSTEM layer.

It is NOT the same as Clusters, which are the Filesystem layer.

Read my citations.

Logical blocks are what the device driver reports to the filesystem driver, clusters or filesystem blocks are what the filesystem manages on it's formatting.

They are completely different things. – Robert Wm Ruedisueli – 2019-12-27T01:08:11.977

Here is another citation from a quick google: https://www.thomas-krenn.com/en/wiki/Advanced_Sector_Format_of_Block_Devices

– Robert Wm Ruedisueli – 2019-12-27T01:11:12.433

Alright, then how do you enable block conversion? You said it's easy but you didn't say how. The question is asking how to change a target drive with 512 byte logical sectors to have 4096 byte logical sectors, so an MBR-partitioned source drive with 4096 byte logical sectors can be copied to the target and used in Windows. – balazer – 2019-12-27T06:13:56.813

I already told you. It's done automatically.

The formatting utility can specify the block size. If it's set to a different size, it's converted by the OS. There is nothing to enable. – Robert Wm Ruedisueli – 2020-01-10T20:39:05.160