How can a MBR formatted hard drive exceed 1.81 TiB capacity?

8

2

I recently purchased an external disk drive with 3 terabytes (2.72 TiB) of storage capacity.

Now according to Wikipedia's article on MBR, the maximum addressable storage space on a disk formatted using the MBR scheme is restricted to 2TB (1.81 TiB) so it stands to reason that high capacity hard disk drives (>2TB) must be formatted using GUID Partition Table (GPT) scheme to be able to use the entire capacity of the HDD.

Why is then the 3TB WD My Book external disk drive factory formatted with the MBR scheme and even more surprisingly, why does it show up on Windows as a 2.72 TiB drive when MBR can only address upto 1.81 TiB?

Has anyone else with a high-capacity Western Digital (or other brand) hard drive seen this before?

If it helps, the HDD is formatted using NTFS and the sector size is 4096 bytes. Also, here are a few screenshots:

3 TiB MBR WD My Book - 3TB


EDIT: After I posted my answer, I read this blog post from AnandTech which says:

Now LBAs under MBR partitions are addressed using 32-bit values, the maximum of which is 2^32 or 4294967296. Each LBA on a hard drive corresponds to a 512-byte sector value (even on 4K advance format drives, they still appear as 512-byte sector drives to the OS), so the largest partition you can have in a MBR partitioned drive is 4294967296 * 512-bytes or 2,199,023,255,552 bytes.

Hard drive manufacturers define 1TB as 1 trillion bytes. If we use that definition then the largest 32-bit MBR partition would be 2.199TB (2,199,023,255,552 bytes / 1,000,000,000,000). If we define 1TB as 1024^4 bytes (TiB) then the largest 32-bit MBR partition would be 2TiB (2,199,023,255,552 bytes / 1,099,511,627,776). Either way, with a 3TB drive there’s no way we’re getting a single 3TB partition using MBR.

But as you can see, the My Book I own has a single 3TB (2.72 TiB) partition and it's using MBR. The reason why this bothers me is because of this incident where a person claims to have encountered data integrity problems with the drive after transferring more than 2TB of data on the MBR formatted 3TB drive.

Vinayak

Posted 2014-12-14T17:49:50.963

Reputation: 9 310

1Is this an Advanced format (aka AF, aka 4K sector size) drive ? – Hennes – 2014-12-14T17:51:46.147

@Hennes Yes, I just found out about that right after I posted the question. That seems to be the reason why this is possible. – Vinayak – 2014-12-14T17:52:53.823

Is this drive 4Kn? 4K native sectors would theoretically allow the MBR to handle 17.6 TB (16 TiB) drives. – bwDraco – 2015-01-18T20:18:41.183

@DragonLord If you meant to ask whether it is an Advanced Format drive, yes it is.

– Vinayak – 2015-01-18T20:22:45.650

I see. However, that article from AnandTech says and I quote "even on 4K advance format drives, they still appear as 512-byte sector drives to the OS" so they're wrong? – Vinayak – 2015-01-18T20:25:42.183

2https://translate.google.com/translate?sl=de&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.heise.de%2Fct%2Fhotline%2FFAQ-3-TByte-Festplatten-1183299.html "Some external 3-TB-drives from Seagate (FreeAgent GoFlex) and Western Digital (My Book Essential) can be used thanks to another technical gimmick on XP at full capacity. Sign up through their USB or FireWire interface not with the usual 512-byte sectors but with 4K sectors to the system. In this way, the 2-TB-limitation of the MBR can be avoided." – bwDraco – 2015-01-18T20:26:55.323

2I'm talking about 4Kn, where the drive is not emulating 512-byte sectors, but is exposing its actual 4K sector size to the system. – bwDraco – 2015-01-18T20:28:16.563

1@DragonLord I just looked up 4Kn and I'm not sure it's a 4Kn HDD. I don't know what hard drive WD uses internally but I suspect it is a WD Green drive. – Vinayak – 2015-01-18T20:28:44.750

I'm composing an answer right now. – bwDraco – 2015-01-18T20:33:32.150

I just found out about fsutil fsinfo ntfsinfo <driveletter:> and fsutil fsinfo sectorinfo <driveletter:> commands from this answer on SO. It reports the physical sector size among other things.

– Vinayak – 2015-01-18T21:08:43.867

I'm totally puzzled with this AF drives issue! I have many WD drives and just not when i bought a 4TB drive i was aware of this... I have a 4TB NTFS drive with ONE full MBR partition and the "WD Align - Powered by Acronis" says the drive IS NOT AF! How can this be?! This post is very informative about the conversion external enclosures can do!!! https://superuser.com/questions/852475/how-can-a-mbr-formatted-hard-drive-exceed-1-81-tib-capacity

– marcolopes – 2019-02-28T20:19:59.520

Answers

2

Your USB drive presents itself to the host computer with a logical sector size of 4096 bytes, and that's what enables it to be partitioned with an MBR partition table that can use more than 2 TB of the drive's space, even in Windows XP. MBR partition table entries are 32 bits, which allows a maximum of 2^32 sectors to be used. 2^32 sectors x 4096 bytes/sector = 16 TB. With 512 bytes/sector, only the first 2 TB of a drive could be used by an MBR partition table. (2^32 sectors x 512 bytes/sector = 2 TB)

However, your drive is not 4k native. There are two layers of translation:

  1. The USB enclosure contains a USB-to-SATA bridge that translates from 4096 bytes per logical sector at the USB interface with the host computer to 512 bytes per logical sector at the SATA interface with the drive inside.
  2. The drive inside internally translates from 512 bytes per logical sector on its SATA interface to 4096 bytes per physical sector. (That's 512 byte/sector emulation or "512e".)

A SMART query passed though the USB bridge to the SATA drive inside reveals the truth (run here on my own Western Digital My Book 3 TB external USB drive in Windows XP):

C:\Program Files\GSmartControl> smartctl -a -d sat pd11
smartctl 6.5 2016-05-07 r4318 [i686-w64-mingw32-xp-sp3] (sf-6.5-1)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD30EZRX-00MMMB0
Serial Number:    WD-WCAWZ12xxxxx
LU WWN Device Id: 5 0014ee 2063xxxxx
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical

The internal SATA drive reports 512 bytes per logical sector and 4096 bytes per physical sector: that's 512 byte/sector emulation, or "512e".

However the drive as seen by the host computer connected by USB has a logical sector size of 4096 bytes:

C:\> wmic DISKDRIVE get bytespersector, caption
BytesPerSector  Caption
4096            WD My Book 1140 USB Device

and that's how Windows XP is able to use all 3 TB of the drive's space.

If you need more proof than that, you could remove the drive from its enclosure and connect it directly to a motherboard's SATA connector: you'd then find that the drive has a logical sector size of 512 bytes, and the drive's partitions would all appear as 1/8 their previous size and be unreadable due to incorrect partition table entries caused by the sector size mismatch. Plenty of people have done this and then had to fix the problem by rewriting the partition table.

I've checked several different external USB drives and they are all 512e internally despite being configured for Windows XP compatibility with 4096 bytes per logical sector at the USB interface: WD My Book 3 TB, WD Elements 3 TB, HGST Touro 4 TB, WD Easystore 8 TB, WD Easystore 12 TB.

Your drive came with an MBR partition table (or someone changed it to MBR at some point) to make it compatible with Windows XP. Newer drives come with GPT partition tables, which require Windows Vista or later.

The WD Quick Formatter tool can be used to change a Western Digital external USB drive between 512 bytes/sector with a GPT partition table (for Vista and later) and 4096 bytes/sector with an MBR partition table (for Windows XP compatibility).

balazer

Posted 2014-12-14T17:49:50.963

Reputation: 86

8

Your drive uses Advanced Format 4Kn, where the drive exposes its native 4K sector size instead of emulating 512-byte sectors (512e). This ensures compatibility with devices that do not support GPT by extending the MBR limit to 17.6 TB (16 TiB), thereby avoiding the need to use GPT.

  • MBR permits a maximum of 232 sectors. This means that with conventional 512-byte sectors, the maximum supported disk size is 2.2 TB (2 TiB). However, with 4K sectors, the maximum disk size becomes 17.6 TB (16 TiB).

  • According to this article (via Google Translate, lightly edited), several external hard drives exceeding 2.2 TB in capacity use 4K native sectors with an MBR partition table for compatibility with Windows XP:

    Certain external Seagate (FreeAgent GoFlex) and Western Digital (My Book Essential) external 3-TB disks can also be used on XP at full capacity thanks to another technical twist. They do not [connect] to the system via their USB or FireWire interface [with the usual] 512-byte sectors, but rather with 4-KB sectors. This bypasses the 2TB limitation of the MBR. But even this trick has drawbacks: programs that access the disk low-level - such as formatting tools - may not work on the disks. [T]hey are [also] not suitable [as boot disks].

  • The manufacturer is taking advantage of the fact that this is an external hard drive, with an assumption that it will only be used for storage and not booting. However, this creates another set of compatibility problems, where programs that rely on 512-byte sector sizes may not work correctly:

    • Partitioning tools that do not support 4Kn drives will not function correctly with this drive.

    • Some database programs use direct disk I/O for performance and data integrity. These programs will not function correctly if they are not designed to handle 4K sectors.

  • The underlying internal hard drive may well be 512e—if this is the case, the enclosure electronics are exposing 4K sectors to the system for compatibility.

bwDraco

Posted 2014-12-14T17:49:50.963

Reputation: 41 701

1While the drive itself is probably indeed Advanced Format, I’m willing to bet it doesn’t really expose its native sector size. Instead, the USB-SATA bridge chip does that. – Daniel B – 2015-01-18T21:11:22.207

1@DanielB: If the drive is 4Kn from the standpoint of the system, it's unlikely the drive is 512e with the USB-SATA bridge converting back to 4Kn. It's much simpler that the underlying drive is 4Kn, and it takes little more than firmware to do this. – bwDraco – 2015-01-18T21:14:09.003

2It might be simpler, but there are tons of enclosures doing all sorts of dark magic to enable 3 TB partitions that also work on XP. – Daniel B – 2015-01-18T21:15:04.847

1@DanielB I ran fsutil fsinfo sectorinfo <drive_letter:> and it reports the logical and physical bytes per sector to be 4096. I'm not going to take it apart and connect it to a SATA port and run the same command again anytime soon, but when and if I do, I'll report that here. – Vinayak – 2015-01-18T21:18:45.813

1

@Vinayak That’s a given, otherwise the MBR partition at this size would not be possible. WD doesn’t offer 4Kn drives though, so it’s unlikely the drive itself is 4Kn.

– Daniel B – 2015-01-18T21:20:32.847

@DanielB I'm not sure if HGST counts but it is a WD company and I found this on Amazon

– Vinayak – 2015-01-18T21:25:42.040

I have a 4TB NTFS drive with ONE full MBR partition and the "WD Align - Powered by Acronis" says the drive IS NOT AF! How can this be?! This post is very informative about the conversion external enclosures can do!!! https://superuser.com/questions/852475/how-can-a-mbr-formatted-hard-drive-exceed-1-81-tib-capacity

– marcolopes – 2019-02-28T20:20:38.030

4

I got in touch with a WD support technician and he told me that it was an anomaly and that I should convert the disk to GPT instead to make sure I don't encounter any accidental data loss in the future.

However, he was unsure about whether or not it would actually cause any data loss but recommended I convert the disk to GPT and he also let me know that mine was the first case of a MBR formatted high-capacity drive (>2TB) that he's encountered.

If you have encountered a similar situation before and know why this is or isn't an issue, please add your answer.

Disregard everything I wrote below. It may or may not be correct.


I think I found the answer. @Hennes and AIDA64 helped. It seems like Western Digital has come up with a solution to prevent the slow demise of MBR and it's called Advanced Format.

With this technology, WD has increased the block size of the HDD from 512 bytes to 4 kilobytes so Advanced Format drives can grow to 16 TB without hitting the 32-bit LBA size limit.

Stephen Foskett has explained this very well in his blog post.


From the blog:

Current hard disk addressing scheme tops out around 2.1 TB. There are two simple factors in this equation:

  1. There are 4,294,967,296 addressable blocks in a 32-bit “namespace”, and this is all most PCs can handle
  2. Each block is 512 bytes in size, and has been since dinosaurs sold the computers

If we multiply 4,294,967,296 times 512, we get 2,147,483,648 KB of capacity. In wacky base-10 storage industry speak, this is 2.15 TB of capacity.

The hard disk drive manufacturers have two roads forward, and they are not mutually-exclusive:

  1. Western Digital was first out with an Advanced Format drive late in 2009. This addresses the (archaic) block size limit, increasing it from 512 bytes to 4 kilobytes. With each address now eight times larger, we need one-eighth as many addresses for a given drive size. Advanced Format drives can grow to 16 TB without hitting the 32-bit LBA size limit.

  2. Seagate attacked the other variable, increasing the address space to the full 48 bits specified in the LBA standard. This gives an amazing 128 petabytes of capacity even with tiny 512 byte blocks.

Vinayak

Posted 2014-12-14T17:49:50.963

Reputation: 9 310

I think I'd prefer Seagate's solution, with lots of tiny files @ 4k each they can take up a lot of space pretty fast – Xen2050 – 2014-12-14T18:44:57.960

@Xen2050 - What? The small files take the same amount of space no matter what. – Ramhound – 2014-12-14T18:51:01.690

1@Ramhound Not when a bunch of <4k files are used on a drive with a block size of 4k, now they each take up 4k. – Xen2050 – 2014-12-14T18:58:58.877

You still get the same amount of data out of the storage device though. Think of it this way. Before a 4kb file would take multiple blocks not it only takes up one. – Ramhound – 2014-12-14T19:08:23.027

@Ramhound So, for a 4KB file, you use 4KB before and 4KB now—no difference. But for a 1B file, you use 512B before and 4096B now—a loss of 3584B. In fact, if we can expect file sizes to be uniformly distributed mod 512 bytes, which seems at least superficially reasonable, then we can expect to lose (4096B/2 − 512B/2) = 1792B per file on average. – wchargin – 2014-12-15T00:09:01.960

What file system are we talking about "we" have solutions to partially used blocks. – Ramhound – 2014-12-15T00:13:43.660

NTFS supports 4k blocks here is how it works http://blogs.msdn.com/b/ntdebugging/archive/2011/06/28/ntfs-and-4k-disks.aspx additionally one has to indicated of its 4k with or without 512b emulation

– Ramhound – 2014-12-15T00:18:17.597

SO, seagate solution was not adopted universally because disk manufacturers wanted to increase capacity by ~10% using 4k sectors? – marcolopes – 2019-02-28T20:23:02.263

@wchargin even if the disk uses 512-byte sector, the file system still uses 4KB cluster in 99.99% of the cases so you'll still end up wasting the remaining 4095 bytes – phuclv – 2019-12-27T12:04:20.613

@Xen2050 see my comment above. Almost no file system uses cluster sizes < 4KB unless you explicitly specify it so you don't waste any more with the new 4K drives. You might avoid the wastage with resident files on NTFS or inline files in ext4 though, which still makes your file takes "0 byte" on disk and you also waste no more in the new drive format – phuclv – 2019-12-27T12:07:06.050

LBA48 addresses a totally different limit (firmware, e.g. (U)EFI/BIOS, communication with the disk controller circuitry that today is typically on the actual disk/storage device), but not partitions. So LBA48 doesn't address or affect a limit imposed by the partitioning scheme (in this case, the MBR partitioning scheme). Option 2, mentioned by the current edit of the answer (from the year 2014), is not relevant to the topic of this question, which is a limit caused by the partitioning scheme that is being used. – TOOGAM – 2020-02-04T13:44:28.813

1

I'm inclined to go with the blog rather than the WD support technician's reply. I have a couple of Toshiba Stor.e Canvio 3TB drives. Both are factory formatted using 'advanced format' with cluster and sector sizes of 4k.

Drives are used on a Samsung smart TV which does not recognize GPT partition (most multi media devices will only recognize MBR).

user409837

Posted 2014-12-14T17:49:50.963

Reputation: 11

That is interesting. Have you used up more than 2 TiB of of storage on any of those HDDs? – Vinayak – 2015-01-18T20:16:19.173

0

4K Advance Format / 512e drives may be one answer, as noted by other answers. I won't duplicate their answers' contents by elaborating too much further. I write this immediately to demonstrate that I don't wish to try to compete with the idea of 4KAF/512e as I do agree with it. However, even using just 512 byte sectors, there is another possibility I see.

The other possibility* is that this could be done with 512 byte sectors on a disk. The actual limit is that the start of a partition needs to be within the first 2 TB**, and then there is another spot in the partition table that specifies how big the partition is. So, in theory, a partition could go from near the 2TB boundary to a location 2 TB later, so MBR could theoretically support partitions up to about the 4TB marker.

I don't ever recall seeing that advertised, and I can imagine why. As data storage devices (hard drives, SSDs) keep growing in size, trying such a method would only help for a single generation of size doubling (between 2TB and 4TB). It would require that people have multiple data partitions (one or more before the 2TB mark, and then only a single one that starts towards the end of the 2TB mark and can not be more than 2TB in length).

Trying to explain this to an average purchaser would likely be a nightmare for a technical support department. It would seem to be in the interest of many manufacturers to just embrace GPT (which sounds like a good solution anyway, in the slightly longer term as drive sizes increase) and not tell people that anything above 2 TB is a layout that they intend to try to commit to supporting.

While I can make no promises of how operating systems will support such a layout, particularly since a lot of people use Windows 10 which can update its behaviors without prior significant notice that most users see, the creators and maintainers of operating systems usually want to do what is simple and sensible. Which means, if a straightforward approach works well, there is no reason to go out of their way to break that. Most operating system development is involving areas of technology that most programmers find to be far more interesting than partition handling. So, I would strongly suspect that if you can somehow, anyhow, create a partition that extends past the 2TB mark, then that partition is likely to actually work just fine. (Especially, if it works in the beginning, it is likely to work after the operating system is installed.) Some partitioning tools might prevent the creation of such a valid layout as some limits may be imposed in an attempt to be user-friendly, but I can think of some partition manipulating software (generally text base) where sector numbers (or CHS values) can be typed in manually, and which are likely to work with that scheme.

[*] I'm writing this answer based on the studying I did on the meaning of the bytes in a Master Boot Record. The question has the "Windows 8.1" tag. I haven't determined if there are any specific behaviors that Windows 8.1 does to limit what a user can get away with during the operating system's installation, or by using "Disk Management" (diskmgmt.msc), or by using DiskPart. So this answer is intended to just address the concept of the 2TB boundary for disks that primarily use the MBR partitioning scheme. If you somehow manage to create such a configuration of partitions, I would imagine most computer firmware ((U)EFI/BIOS) and most operating systems will happily boot from, and effectively use, it. Also, I imagine that most operating system installation procedures will probably happily use such a partition if it pre-exists. Objecting would require specific checks that aren't likely to provide much benefit, so such checks probably aren't widely done.

[**] Quoting from my own website, "The latest sector that could start a partition would be sector 4,294,967,295 which is byte 2,199,023,255,040, which is 512 bytes short of the 2TB mark."

TOOGAM

Posted 2014-12-14T17:49:50.963

Reputation: 12 651