Full formatting, HDD controllers, and bad-sector locking

Question

We get periodic questions about manually locking and reassigning bad sectors on hard disk drives. There seems to be apparently conflicting information on one aspect of this, the hard drive controller and full formatting.

Hard Drive Controller

Hard drive controllers perform some automated recognition of bad sectors, lock (mark) them, and automatically reassign the contents. This is reported in the SMART information. My understanding is that this is a "random" process, i.e., it happens when the controller stumbles across a bad sector during drive usage.

Full Format

The accepted answer on this question, How to isolate bad sectors on a hard disk in Windows 7?, quotes Microsoft in describing a full format. It talks about the process including a scan for bad sectors, and that in Vista and later, this is accomplished by writing zeros to the entire partition.

Implication

The implication seems to be that the full format uses the drive controller's automated process, and the zero-fill provides a means for the controller to examine every sector. It's possible that sector verification is controlled by the format utility, itself, but this is not described.

No Inherent Read Verification

In a recent question, How do Zero-Fill, Full Format, and CHKDSK fix bad sectors?, there is discussion of this in comments on the answer. In this comment, user sawdust states,

Zero-fill or "formatting" for a filesystem use ordinary write operations. There's no automatic verify or read after write, but there would have to be a read in order to detect a bad sector. So I would not expect any change in the bad sector lists maintained by the controller.

My attempts to research this issue did not uncover any articles on bad sector cleanup that mentions using a full format as a means to accomplish this, which would be consistent with sawdust's comment.

Which brings me to this question, and these two components:

• How does a full format identify bad sectors? i.e., is the bad sector identification (read verification) managed by the controller as part of its automated response to the zero fill, or does the format utility perform reads of each sector?

• What condition triggers the hard drive controller's automated process to lock a sector and reassign the contents? i.e., if writes don't inherently involve read verification, is the action triggered only by an inability to correctly read a sector (at least on the initial try)?

• And corollary: if the format utility performs its own read verification, which I assume would be against the zero-fill (and then marks an external bad sector list, as described in cybernard's answer), wouldn't that read action also trigger the automated verification by the controller? i.e., wouldn't that create redundant checks, and so why would the external list be needed?

Answer 1

1. A full format, simply hides the bad sectors. Most file systems have a bad block list, and it simply adds the bad sector marker in the file system appropriate way to indicated do not use me.

2. Hard drives have SMART built-in. Self Monitoring And Repair Tool. Each sector has an ecc(error correcting data). Eventually, if the physical hard drive has a hard enough time reading a sector it will replace the bad sector with a spare. The spares are typically very limited, maybe 100-300 on most drives. When those are used SMART throw off a warning your drive is about to fail. This list hides the bad sectors even from a format.

When writing it does verify the results, and write new ecc data. However, a full format doesn't write to the whole drive only the file system structure. It simply reads the whole drive.

An error is detected by non-matching ECC, duration it takes to read the sector, and other internal measurements. One reason the format utility needs to mark bad sectors is because the physical drive only has 100-300 spare. Another reason is because the format utility was not programmed with the ability to talk to SMART and tell it a problem exists.

Only Windows 7 and above have any ability to communicate with SMART, and even then many times that ability goes unused.

One major caveat, here hard drive manufacturers didn't want SMART and were forced to implement it, it memory serves correctly via lawsuit. However, in many cases the manufacturers have limited its usefulness. SMART is supposed to upon read/write failure a certain amount reallocate the sector, but I have found through real world experience that either the threshold is insanely high or ignored.

There are very few programs that can force the hard drive to re-allocate the bad sectors mhdd and spinrite, but even then I have seen them fail to re-allocate many sectors.

Answer 2

How does a full format identify bad sectors? i.e., is the bad sector identification (read verification) managed by the controller as part of its automated response to the zero fill, or does the format utility perform reads of each sector?

Who claims that a "full format" is supposed to "identify bad sectors"?

I've performed a few timing tests, and the results indicate that a Windows 7 format only writes the sectors.
You could have a HDD that has incorporates this patent which is implemented by the optional Write-Read-Verify feature set. Use of Write-Read-Verify could be detected by the performance degradation of the extra rotation required per operation.

Without a read operation any errors in the data record of the sector will not be detected.

What condition triggers the hard drive controller's automated process to lock a sector and reassign the contents? i.e., if writes don't inherently involve read verification, is the action triggered only by an inability to correctly read a sector (at least on the initial try)?

"Lock a sector" is not conventional terminology.
Bad sectors are remapped, and described in this article.
The primary defects are remapped during manufacturing.
Defects that accumulate during the life of the drive are remapped by the controller and maintained in other lists. The criteria for remapping may vary by vendor, the minimum being repeated uncorrectable (read) errors. Uncorrectable errors in the Identification record of a sector would be immediate cause for remapping.

And corollary: if the format utility performs its own read verification, which I assume would be against the zero-fill (and then marks an external bad sector list, as described in cybernard's answer), wouldn't that read action also trigger the automated verification by the controller? i.e., wouldn't that create redundant checks, and so why would the external list be needed?

Again, I find no evidence that Windows 7 explicitly performs anything but write operations during the format of a partition.
There's nothing unique about a zero-fill. It just happens to be the default data when a low-level format is performed. The ECC is capable of detecting (and correcting) an error burst of multiple bits.

What you think is redundant is due to circumstance.
Older disk controllers lacked the processing power and internal storage to perform bad-block maintenance.
Is was left to the OS/filesystem software on the host computer to manage bad blocks.

With the advent of zoned-bit recording, the format command was removed from the ATA4 command set. The elimination of this format command necessitated that the HDD controller assume some of the bad-block maintenance.
Actually this was a win for the HDD manufacturers, as the primary defect lists would be hidden and platter yields could be increased by accepting more defects.

So bad-block maintenance by the HDD controller can reduce costs for the manufacturer. This can reduce some complexity in the OS/filesystem, but could impact performance (i.e. an additional seek).
Bad-block maintenance by the OS/filesystem typically just prevents allocation of the bad sectors. So there's no performance hit like a remap, but the entire allocation unit (aka cluster) of each bad sector has to be unallocatable.
The OS/filesystem cannot eliminate its bad-block maintenance because of the need for backwards compatibility for older HDDs that are not ATA4 or newer.