25

If the goal is to make data no longer retrievable, how secure is it to format the disk? I assumed formatting the disk overwrites free space (thus making it a safe bet no one's going to be able to retrieve the data) but according to webopedia this is not the case. How much more secure is it to delete files with an erasure utility (using something like Schneier's method)than formatting the drive? How does formatting the drive not erase all data: formatting involves recreating the file system so this seems to imply the data is not retrievable.

I generally do not choose the "quick format" option.

schroeder
  • 123,438
  • 55
  • 284
  • 319
Celeritas
  • 10,039
  • 22
  • 77
  • 144
  • 1
    Note that the answer is very different depending on whether it is a conventional hard disk (spinning platters) or an SSD. – sleske Apr 28 '16 at 07:00
  • Related: [How can I reliably erase all information on a hard drive?](http://security.stackexchange.com/questions/5749/how-can-i-reliably-erase-all-information-on-a-hard-drive) – sleske Apr 28 '16 at 07:01
  • 1
    Maybe note that the so called "Schneier's method" is in fact based on a misunderstanding. Originally, experiments were conducted to see which of several methods would prove beneficial in wiping data. This included writing all 0's, writing all 1's, as well as writing random data. The conclusion was that writing random data two or more times is absolutely sufficient. The original paper was subsequently often misinterpreted as stating that *all* examined methods should be carried out in sequence. – JimmyB Apr 28 '16 at 11:46

4 Answers4

48

Quick-formatting a hard disk simply erases the filesystem's structures and tables and writes new ones in place, giving the illusion of a brand new disk. Old data is simply overwritten as and when needed, but it still remains on the disk. File carving utilities can go through the disk data and recover fragments of files, then stitch them back together without needing the original filesystem entries. This is commonly offered in commercial "undelete" applications, but more comprehensive methods are available in forensics packages.

Wiping a disk with a single pass of random data (or zeroes, or whatever really) is sufficient to fully remove all traces of the data from the overwritten sectors. Multiple passes are pointless on modern disks, even against the perceived threat of hardware-level recovery attacks. I refer you to this question for details, but the short answer is that old techniques like magnetic force microscopy (MFM) were never really effective at recovering overwritten data in the first place on low-density devices, and newer magnetic disks have such high densities that it's physically impossible. Multi-pass overwrites are there to help people validate their need for über-security, or sell magic disk wiping software, despite it being pointless and detrimental to disk longevity.

The only exception is flash (e.g. USB flash drives and SSD), which have additional wear-leveling sectors to increase the lifespan of the device. The physical sectors are exposed as a logical map to the system, which makes it impossible to directly overwrite all of the data. Even if you overwrite all the logical sectors, old data might remain in the wear-leveling sectors. In order to combat this, some flash device specifications include an encryption requirement to increase the difficulty of recovery (because it is no longer possible to directly read data from the memory chips using hardware probes).

In SSDs, encryption can be used to speed up the ATA Secure Erase feature. All sectors on the disk are encrypted using a key stored in hardware, and this key can be discarded and a new one generated when a Secure Erase command is sent to the device, thus rendering all data on the disk (including slack / wear-leveling areas) unreadable. Secure Erase is also possible without encryption, but then the drive must actually delete all memory cells, which may take a while.

Community
  • 1
Polynomial
  • 132,208
  • 43
  • 298
  • 379
  • 8
    **"(...) rendering all data on the disk unreadable"** - the old data is still there, but encrypted. If the discarded key can be cracked (yeah, hundreds of years) then the data is available agian. – Mindwin Apr 27 '16 at 16:08
  • 6
    What if the quick-format option is not checked? – Celeritas Apr 27 '16 at 19:28
  • @Celeritas On Windows when a full format is performed, the OS merely *reads* the entire storage device to check for bad sectors. It doesn't *write* any more data than the quick format option. – Nayuki Apr 28 '16 at 04:14
  • 2
    Note that spinning disks perform sector reallocations, whilch have effects similar to the wear leveling of SSDs. So overwriting all sectors of the disk does not erase all data, because of reallocated sectors (and the HPA). Still, it will typically erase 99% of the data. The safest option is SATA Secure Erase. – sleske Apr 28 '16 at 08:04
  • Also, encryption is not required for Secure Erase, it just makes it faster. I submitted an edit. – sleske Apr 28 '16 at 08:10
  • 1
    @Nayuki I doubt that statement. Do you have a source for it? – JimmyB Apr 28 '16 at 11:40
  • As @Mindwin said, "Secure Erase" is a misnomer. It's not erased until it's erased. Leaving encrypted data lying around is not the same as erasing the data. Period. – James Apr 28 '16 at 14:01
  • @Nayuki I've never heard that either. My understanding is it overwrites each sector. – Polynomial Apr 28 '16 at 15:34
  • @Mindwin Since 256-bit keys are [physically impossible to bruteforce](http://crypto.stackexchange.com/questions/1145/how-much-would-it-cost-in-u-s-dollars-to-brute-force-a-256-bit-key-in-a-year), I wouldn't worry about it. – Polynomial Apr 28 '16 at 15:36
  • @JimmyB Try it yourself. Use a hex editor to read your disk. Do a "full format". Then read the disk again. – Nayuki Apr 28 '16 at 16:37
  • 2
    @Polynomial (...) as of now? yes. Tomorrow? XXII century? Moore's law? New math? New computers? who knows. And I stated **"hundreds of years"** in my comment above. Nobody is worrying, but since this is a theoretical discussion, we have to consider all possibilities. – Mindwin Apr 28 '16 at 17:35
  • 2
    @Mindwin No, 256-bit keys are physically impossible to brute-force due to the energy required; a classical computer would require more energy than the universe contains, and even with Grover's algorithm a quantum computer would require 2^128 operations and a large gate cost, which still requires almost as much energy as we can observe in the known universe. It is physically impossible from a brute-force perspective. Judging by cryptographic attacks from the past hundred years, feasible ciphertext-only exploits are far and few even with seriously weaker designs than AES. – Polynomial Apr 28 '16 at 18:44
  • 1
    @Polynomial I know all that. But I will still bet on future. It is rather depressing to think that is a dead end, a closed case. Just look at our knowledge of physics 20 years ago and now. Think how it will be 20 years from now. Lets put this discussion on hold, shall we? I already said **I agree with you it is impossible today**. We can retake this argument in 20 years. Seems fine by me. – Mindwin Apr 28 '16 at 19:50
  • @Mindwin Then take the same approach with TLS, password hashing, and everything else. Your position is ridiculous from a practical perspective. *Might* AES be broken in future? Sure. *Might* a meteor wipe the planet out? Sure. But within the foreseeable future it's fine, nobody needs to panic, and planning / compensating for it is a complete waste of your time on this planet. Feasibly speaking a discarded key renders AES-encrypted data unreadable. Your data is, in all practical senses, gone. – Polynomial Apr 29 '16 at 14:20
  • 4
    @Nayuki Apparently since Vista, full formatting does a complete wipe, see e.g. https://support.microsoft.com/en-us/kb/941961 – JimmyB Apr 29 '16 at 15:00
  • This answer doesn't seem to answer the question. The question was asking if a full format is secure. The answer starts by talking about quick formatting, which the question explicitly isn't about. The answer then goes on to talk about wiping disks without saying whether or not a full format does that. We're left not knowing whether or not a full format is secure. But the latest comment by @JimmyB does seem to answer that. – Kyle Delaney Dec 02 '21 at 04:05
4

One part of your question seems not to have a good answer, so I want to add to this:

By formatting a drive, you typically choose to do a "quick format". This does not overwrite the data, but only clears the information where each file is stored. Therefore, it is as easy as starting a program to recover a hard drive that has just been "quick formated".

Maybe this comparison helps you to understand this better: Your hard drive is like a book - and the partition(s) and the file system(s) are the table of contents. A quick format only erases the ToC. This is faster than erasing each single page - but the information in the book stays recoverable.

At least for classical hard drives, overwriting each single sector should provide enough security for most cases.

Lukas
  • 3,138
  • 1
  • 15
  • 20
  • 2
    To add a quick note to the metaphor - a quick format removes the Table of Contents of the book, allowing you to write a new Table of Contents later which overwrites the existing content of the book. But after only a Quick Format, a data recovery program can easily still read the book. Once you overwrite the data, not so much. – Jake Apr 27 '16 at 16:56
  • 1
    @Jake yes exactly! – Lukas Apr 27 '16 at 21:35
1

If "disk" happens to mean "solid state disk", you are somewhat lucky. You can instantly make retrieval of data (almost) impossible by using the manufacturer's secure erase tool. This process is both very fast and secure due to the way these drives work: They encrypt all data that is written to the drive, no exception made. Doing a "security erase" simply tosses the key overboard, and the data, while still present, is unrecoverable without the key.

Otherwise (i.e. "disk" means "harddisk"), the only safe method of making data unrecoverable is by disassembling the device and physically destroying the platters.

The reason for that is two-fold. First, overwritten data can still be read. Data density has admittedly gone up considerably during the last decade, and some people argue that what's overwritten once is basically unrecoverable, but if it is really, really important, I wouldn't bet my right hand onto that. I haven't tried to recover erased data during the last 15 years, but it used to be that recovering 5-6 times overwritten data was kinda tedious, but otherwise absolutely no problem. Maybe that's different now, maybe you only need to overwrite twice, or maybe once. But there's a bigger problem ahead.

The second, and much more important issue is that you don't know if and when you overwrite something. This is a very serious problem.

Modern drives do transparent wear-levelling and sector reallocation, and maybe even caching on MCL (hybrid drives). You have absolutely no knowledge, or control, what gets written when and where, or what gets overwritten.

Thus, even a "secure erase" that overwrites the same file ten times with different patterns might indeed overwrite ten different sectors on the disk, none of them belonging to the original file (and you might possibly still have an old copy of the data stored in a retired, now inaccessible block).

The only way of knowing for sure is disassembling the drive, putting a neodym magnet onto the platters for a few minutes, and giving the board and platters a little treatment with the hammer afterwards.

Damon
  • 5,001
  • 1
  • 19
  • 26
  • 10
    "*It used to be that recovering 5-6 times overwritten data was kinda tedious, but otherwise absolutely no problem.*" From what I've read, this is more of an urban legend than fact, with no more support than a single researcher's *theoretical* claims to support it. Do you have any references to support it ever being "tedious but no problem" to recover data from a single-pass randomized overwrite? This is the best resource I have found supporting the opposite, but I'm happy to be proven otherwise: http://web.archive.org/web/20121110053501/http://grot.com/wordpress/?p=154 – loneboat Apr 27 '16 at 18:34
  • 1
    There are several mistakes here: Not all SSD drives use encryption; spinning platter disks _can_ be (quite) safely erased without destruction (ATA Secure Erase); "it used to be that recovering 5-6 times overwritten data was kinda tedious," - I don't think this has _ever_ been done. etc. etc. – sleske Apr 28 '16 at 08:12
1

Windows, since Vista and including all editions of Windows 10, overwrites the whole drive with zeroes if the "quick format" option is unchecked.

Source: Microsoft docs

(Thanks JimmyB for posting this link in the comments.)

Matthias Braun
  • 421
  • 3
  • 12
Felix Dombek
  • 111
  • 3