102

As storage technologies change over time, using different encodings and remappings to deal with sector errors, the best way to permanently erase/wipe/shred data changes also.

Methods for flash drives and other solid-state drives are covered nicely at Jesper Mortensen's answer on SSD erasure.

Various methods for hard drives are described at http://en.wikipedia.org/wiki/Data_erasure and include

Does the ATA "secure erase" cover all modern hard drives and interfaces?

Are there modern storage devices besides SSDs that require something different?

The other option I've heard of is always encrypting everything you send to the drive yourself via something like Truecrypt. If you do that, it would seem that you have to be able to securely destroy the keys when necessary. And in the absence of a neuralyzer, does human memory of the password render any hard disk encryption schemes vulnerable to data recovery?

nealmcb
  • 20,544
  • 6
  • 69
  • 116
  • 2
    Maybe you should break this up, this covers, IDE, SCSI, SATA, and SSD. These drives use different recording methods, densities, and controls. For example a low density IDE drive may be recoverable if track fluxes have wobble, but high density SATA drives are not. – this.josh Jul 29 '11 at 07:55
  • 1
    @this.josh - I intended to cover non-SSDs - that's why I split SSD out by referring to the other answer. From what I've heard so far, there is enough similarity in how the others should be approached that it makes sense to cover them here - e.g. if secure erase doesn't work for a particular drive then fallback to a simple overwrite of the raw disk should work. If there is a better phrasing for that than "hard drive" (rotating disc?), let me know. – nealmcb Jul 29 '11 at 20:22
  • 1
    What matters is remence of data after rewiting. I can think of two factors that may cause remence: difference between the area changed by the write head and area sensed by the read head, and linear and track precision. As linear density and track density increase there is less area for remence due to imprecision. There is an order of magnitude difference between drives manufactured in 2001 and 2005. If you go back to 2001 densities, could a exceptionally well resourced atacker find remence? – this.josh Jul 29 '11 at 22:13
  • 1
    @this.josh Right - "it depends". So I guess as I see it, we might as well have a single question for hard drives, since a good answer will inevitably need to give a risk-sensitive process for deciding what to do, and there is enough similarity in the various hard drive technologies and answers that having them in one place is useful. E.g. depending on the threat level and technology, people will want to know how to decide whether a single overwrite is suitable or whether one of Gutmann's methods would be cost-effective. – nealmcb Jul 29 '11 at 23:23
  • Related: [Why is writing zeros (or random data) over a hard drive multiple times better than just doing it once?](http://security.stackexchange.com/q/10464) – Gilles 'SO- stop being evil' Jan 08 '12 at 01:25
  • 1
    Related: [How do you destroy a hard drive](http://security.stackexchange.com/questions/11313/how-do-you-destroy-an-old-hard-drive) – makerofthings7 Oct 16 '13 at 14:03
  • 1
    See also: [How do you destroy a CD or DVD safely](http://superuser.com/q/660221/47507) on SuperUser – makerofthings7 Oct 16 '13 at 14:06
  • A shredder works very reliably: https://www.youtube.com/watch?v=LsX1GnC8FQQ – Jesse K Sep 07 '16 at 17:15

8 Answers8

41

The only NIST approved method to securely erase a hard drive is by utilizing the secure erase internal command - documented at the Center for Magnetic Recording Research (CMRR) - and that is what everyone should be doing. It is an ATA command, and covers (S)ATA interfaces.

After that, you can optionally degauss the drive to erase the firmware itself.

Lots of interesting info in Guidelines for Media Sanitization, NIST SP 800-88 (2014).

A quote from the paper:

Secure Erase

An overwrite technology using firmware based process to overwrite a hard drive. Is a drive command defined in the ANSI ATA and SCSI disk drive interface specifications, which runs inside drive hardware. It completes in about 1/8 the time of 5220 block erasure. It Was added to the ATA specification in part at CMRR request. For ATA drives manufactured after 2001 (Over 15 GB) have the Secure Erase command and successfully pass secure erase validation testing at CMRR. A standardized internal secure erase command also exists for SCSI drives, but it is optional and not currently implemented in SCSI drives tested by CMRR. SCSI drives are a small percentage of the world’s hard disk drives, and the command will be implemented when users demand it.

A more recent (2010) short presentation from NIST that compares different erase techniques and their limitation is in a ppt presentation by Lyle and Russell.

Matthias Braun
  • 421
  • 3
  • 12
john
  • 10,968
  • 1
  • 36
  • 43
  • 2
    Thanks. Also, you might specifically note that the HPA (http://en.wikipedia.org/wiki/Host_protected_area) and DCO parts of the disk would be missed by normal software using the OS to erase the drive. But ATA secure erase will generally still erase them - see [Accessing HPA and DCO Areas on Hard Drives « Data Destruction Topics](http://destructdata.com/blog/?page_id=282). – nealmcb Jul 29 '11 at 05:07
  • 1
    I Noticed on this page https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase about secure erase that it mentions the hdderase command you linked to but does say that "controller support is spotty at best" so perhaps one to be aware of – Rory McCune Jul 30 '11 at 10:55
  • 2
    Also from http://www.consulvest.com.au/UserFiles/Page/UserPages/hdd_sanitisation.php you need to watch your tech using secure erase as "although SCSI, SAS and Fibre Channel drive specifications classify the Secure Erase Implementation as "optional". To date it is unknown if any SCSI, SAS or Fibre Channel drive to support the Secure Erase technology. " – Rory McCune Jul 30 '11 at 10:59
  • @nealmcb Yes, I should have mentioned that. That is one of the most important uses of secure erase, and the reason most forensics people used it in a lab I used to be. – john Aug 01 '11 at 14:36
  • @RoryMcCune From my experience, most of today's controllers support the normal secure erase command. I guess support for the enchanced version is not that good. But you can try it yourself at your disk, just start the program and it will report if it's supported or not. Don't worry, there are confirmation steps before the command :-) Now on the drives side, I have noticed that there are issues with the support from a lot SSD drives. – john Aug 01 '11 at 14:40
  • 1
    The first link documenting the secure erase internal command has gone dead. [Here](http://archive.is/KQO8) is a web archive to the link, but most links on that page are dead. – Bacon Bits Jul 19 '17 at 20:33
30

From a theoretical standpoint the idea of total drive destruction may be the only way of destroying data on a hard drive fully.

From a practical standpoint, I've not seen any evidence that it's possible to recover meaningful data from a standard hard drive (ie, not taking SSDs or other devices that use wear levelling or similar technologies) after a once over wipe from /dev/zero or similar.

There's an interesting article here, that goes into some depth on the opinion that a single pass wipe is sufficient.

Some good additional information from the links provided by @woliveirajr seems to confirm that point. From This paper

This study has demonstrated that correctly wiped data cannot reasonably be retrieved even if it is of a small size or found only over small parts of the hard drive. Not even with the use of a MFM or other known methods. The belief that a tool can be developed to retrieve gigabytes or terabytes of information from a wiped drive is in error. "

And from this document on the subject of using MFM for retrieving data from a disk

Extrapolating this (and forgetting that an 80GB drive from 2006 does not compare in the resolution requirements of a 1Tb drive from 2008 ) it would be expected that a complete image of a 1 TB hard disk platter would take around 89 years to completely image using an MFM based technique.

Update: Another interesting answer on this topic from the Skeptics site here, has some good information and links on the subject of single pass wipes being sufficient.

Rory McCune
  • 60,923
  • 14
  • 136
  • 217
  • 1
    I'd think a once over wipe would miss remapped sectors, journal data, etc, which is why there is a specific ATA secure erase command. According to the SSD answer I cited, it also is inappropriate for flash drives. So that article seems off base. – nealmcb Jul 28 '11 at 15:21
  • 5
    well remapped sectors only occur (AFAIK) when the sector is marked as unreadable, so it could be argued that no usuable data is present. For anything higher level than that (eg, File system journal data) a command like DD would (again AFAIK) overwrite everything if done on the entire device. – Rory McCune Jul 28 '11 at 15:29
  • dd would also miss HPA and DCO sections - see my comment on John's answer. – nealmcb Jul 29 '11 at 05:09
  • 1
    interesting. From the wikipedia article it looks like user data is not typially stored in the HPA, so for the purposes of reliably deleting your data from a disk would it matter that it's not covered? – Rory McCune Jul 29 '11 at 06:47
  • It would depend of course on your threat model, which might include e.g. destroying PII that Computrace might put there to load software that reports to their servers whenever the machine is booted on a network, or rootkits that secreted data away there. But if a NIST-approved method that is 8 times faster also deals with these questions, it sounds good to me. – nealmcb Jul 29 '11 at 19:36
  • 3
    One thing on the secure erase though, aren't you reliant on the controller manufacturer having correctly implemented the command? From https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase it mentions that " controller support is spotty at best." for the hdderase utility – Rory McCune Jul 30 '11 at 10:54
  • Interesting. I did wonder about that sort of thing. I wonder if the NIST paper addresses it. Couldn't hurt to do both, though it would be a bit slower. Some benchmark data or timing hints would be nice. – nealmcb Jul 30 '11 at 16:37
  • @Rory McCune: A little clarification; the "spotty" controller support reference on the ATA Kernel Wiki refers to the DOS based HDDErase tool, and especially DOS's compatibility with modern motherboards. In contrast Linux HDPARM has good support, because of the modern Linux kernel drivers. In all cases, once initiated, the actual ATA Secure Erase is carried out by the *harddisk alone (on-disk controller)*. So we are mainly/only relying on the harddisk manufacturer to have implemented ATA Secure Erase properly; and according to the CMRR paper linked by john and others that is a safe assumption. –  Aug 01 '11 at 21:41
  • 2
    well each to their own on this one I guess, I'm a bit leary of a claim that this is always implemented correctly given the known problems the the implementation of it on SSDs (eg, http://www.usenix.org/events/fast11/tech/full_papers/Wei.pdf) whereas dd is an extremely well known unix command that would (for most purposes assuming you don't need unwriteable areas covered) handle the job :) – Rory McCune Aug 02 '11 at 06:44
  • dd and similar tools are especially bad for SSD drives, as these drives constantly remap internally their blocks for wear leveling - and have about 10% extra hidden capacity to do just that. In forensics labs, after wiping with dd, you will find user data in this 10% extra space, which the controller did not let dd access. (of course to access that you need to reprogram the firmware or change the actual chip, but these things do happen in forensic investigations) – john Aug 05 '11 at 16:03
  • sure i wouldn't recommend dd for SSDs, but then with the problems with secure erase on SSDs as well, it would seem that physical destruction is the only surefire way to go (well unless you're confident about the secure erase software for a given drive) – Rory McCune Aug 05 '11 at 17:53
14

As storage technologies change over time, using different encodings and remappings to deal with sector errors, the best way to permanently erase data changes also.

Very smart people have expended enormous amounts of time and effort arguing over this problem. Most of them end up at the same bottom line, which is: the only method you can truly trust is physical destruction of the media. (And then those very smart people tend to argue over what sorts of physical destruction are sufficient). Keep this in mind as you start worrying about newer types of media (SSDs) which more blatantly fail the old "rules of thumb" for soft deletion of data.

The other option I've heard of is always encrypting everything you send to the drive yourself via something like Truecrypt. If you do that, it would seem that you have to be able to securely destroy the keys when necessary.

Yes, that is a viable method, and you're right about the keys. I would add: if you're paranoid enough to use this method, you're paranoid enough to have to worry about whether your opponent has the ability to compromise your (machine, keyboard, cables) in such a way as to defeat this method.

And in the absence of a neuralyzer, does human memory of the password render any hard disk encryption schemes vulnerable to data recovery?

If the keys exist, then yes, the human memory of the password becomes the sticking point. And humans often will disclose passwords under the threat of physical/emotional/financial harm, physical/emotional/financial enticement, subpoena, or 'contempt of court' jail sentences.

gowenfawr
  • 71,975
  • 17
  • 161
  • 198
  • 1
    Shredding or melting the platters will work. (Well, anything that heats the platters above the http://en.wikipedia.org/wiki/Curie_point destroys their magnetic alignment (which is used to record the data), so that should work) – Billy ONeal Jul 30 '11 at 06:06
  • 4
    Aparently shredding is not a guarantee. "physical destruction is not absolute if any remaining disk pieces are larger than a single record block in size, about 1/125” in today’s drives" [Center for Magnetic Recording Research 2008](http://cmrr.ucsd.edu/people/Hughes/documents/QandAforwebsite10212008_000.doc) But recovery looks practically impossible. "recovering any actual user data requires overcoming almost a dozen independent recording technology hurdles." [Center for Magnetic Recording Research 2008](http://cmrr.ucsd.edu/people/Hughes/documents/QandAforwebsite10212008_000.doc) – this.josh Aug 01 '11 at 18:20
13

Do you need to erase the data, or do you need to persuade other people that the data has been erased?

(I will only talk about 'entire disk' wiping on conventional drives; I'm not talking about wiping single files or slack space or SSDs.)

As far as I am aware there is no software package that claims to be able to recover data that has had a single overwrite. There are no companies that claim to be able to recover such data. There are no research papers that have shown such data recovery. So, for most purposes (risk:cost analysis) a single overwrite of all 0 is fine, although a couple of overwrites of pseudo-random data is probably better.

But then you may have to persuade auditors that the data has gone. And maybe, just maybe, there is a super secret method used by TLAs to get that data. If your task is to persuade other people (auditors, customers) that the data is unrecoverable and will remain so for ever a secure ATA erase followed by mechanical shredding do.

DanBeale
  • 2,064
  • 3
  • 18
  • 27
  • A brief comment about Wikipedia: WP is about verifiability, not truth. Their "reliable sources" policy means that 'someone can use $TECHNOLOGY_X to recover data over-written just once' (an idea which appears in many places) will appear in the article, even if it's not strictly true. – DanBeale Jul 30 '11 at 17:49
  • 1
    "the multiple overwrite approach is not very much more effective than a single overwrite since it does not do much to the remaining track edges where most of the very low level distorted remnant data remains after an overwrite and it takes a lot more time"[Center for Magnetic Recording Research 2008](http://cmrr.ucsd.edu/people/Hughes/documents/QandAforwebsite10212008_000.doc) – this.josh Aug 01 '11 at 18:26
  • 1
    You wrote: "a single overwrite of all 0 is fine, although a couple of overwrites of pseudo-random data is probably better". Is there a reason why it takes "a couple" of random writes? I would expect a single random write is better than an all-zero write. – H2ONaCl Mar 17 '18 at 05:01
8

If ATA Secure Erase is not an option or not supported on your hard drive, I'd recommend DBAN. DBAN is a well-engineered piece of software for wiping a hard drive, by performing multiple overwrite passes at a very low level. As a result, it is time-consuming but very thorough. It includes methods based upon Gutmann's research, DOD standards, and other research -- and it is easy to use.

Caution: DBAN is only suitable for use with hard disks (magnetic storage). It is not appropriate for flash-based storage, such as SSDs.

DBAN does not erase the Host Protected Area (HPA) (often used to store a recovery partition for your OS, and thus probably relatively low-risk). DBAN does not erase remapped sectors (i.e., blocks marked as bad) unless you specify a particular option. I would expect remapped sectors to be rare. DBAN does not erase the Device Configuration Overlay (DCO) portion of the hard disk (stores hard disk configuration; I would expect it to be relatively low-risk in most settings). If these are a concern, use ATA Secure Erase. Despite that, I expect DBAN to be good enough for most users with magnetic hard drives.

For relatively modern hard drives (ones that support ATA Secure Erase), I recommend using ATA Secure Erase. But if, for whatever reason, ATA Secure Erase is not available (e.g., you have a SCSI drive; you have an older hard drive that doesn't support ATA Secure Erase), DBAN is probably more than adequate.

Be very careful when using DBAN! It will irreversibly destroy your data.

D.W.
  • 98,420
  • 30
  • 267
  • 572
  • So why does NIST specify Secure Erase instead? Perhaps because DBAN misses reassigned blocks, the HPA and DCO areas, etc? See e.g. http://destructdata.com/blog/?page_id=282 – nealmcb Aug 01 '11 at 17:30
  • 1
    nealmcb, I can think of two reasons. (1) DBAN is open source. NIST very rarely recommends any open source projects due to the inherent volatility of that type of project. Furthermore they wouldn't risk introducing untrusted software to an organization. (2) DOD 5220.22-M, the good old 7 pass technique (re: DBAN), relies on the a process of standardizing residual magnetic signature on the platters, to minimize the ability to reconstruct the data. This works on physical discs but not so well on other media. – grauwulf Dec 04 '12 at 16:59
4

Hard drive destruction is not that easy. There's some theory over it, metal just loses the magnetic data when heated above certain level.

If you do a good research on google you'll find some theory about recovering HDD information using powerful microscopes and their magnetic signal.

Encrypting data before storing is the best solution, if you:

  • use good cryptography

  • use protection against someone stealing your key (by physical access to your computer, for example, intercepting your keyboard, memory, etc)

  • have time to securely erase your key when throwing away the HDD.

And that might not be that easy. Secure your computer means, for example, that no one can access it even just after you have secure-deleted-and-wiped the key.

Password is somehow related to the key. If you delete the key, the password is kind of useless. So a combination of good key + good password, losing one make the other useless.

A good combination in truecrypt is to use more than one key, each stored in a different media. For example, one in the HDD, one in the flash memory, another in a token.

Use the effort that you need according to the security you want in your information.

Finally: have you downloaded the source code of truecrypt, read it, compiled your own version? If not, how can you trust it is secure? :)

woliveirajr
  • 4,462
  • 2
  • 17
  • 26
  • 3
    Do you have any citations for the idea that magnetic force microscopy will find any data from a modern hard drive? I've heard it said many times, but not seen the data to confirm it. – Rory McCune Jul 28 '11 at 17:24
  • here, take a look at the bibliography mentioned in the end of this page: http://computer-forensics.sans.org/blog/2009/01/28/spin-stand-microscopy-of-hard-disk-data/ – woliveirajr Jul 28 '11 at 17:35
  • @Rory one PDF: http://www.vidarholen.net/~vidar/overwriting_hard_drive_data.pdf – woliveirajr Jul 28 '11 at 17:40
  • 2
    Thanks for those papers, very interesting. But reading both of them seems to confirm what I was saying that in practical terms data cannot be retrieved once overwritten one. – Rory McCune Jul 28 '11 at 19:26
  • @Rory : in general, I agree with you. Depending on the information and it's value, I'd choose my approach. I'll update my answer to exemplify. – woliveirajr Jul 28 '11 at 19:58
  • As far as cryptographic solutions go, it's also worth remembering that data encrypted today will probably be significantly easier to decrypt in 10 years time, which may or may not matter. – Phil Lello Mar 18 '16 at 14:58
2

When evaluating your erasure method, you need to consider:

  • How quickly you need to erase the data (Is it an emergency response to terrorists banging on your door, or are you selling your old hardware on Ebay next week?)
  • How much effort, money, or time will any potential attacker realistically spend trying to get at your data? (Are you a high-stakes target, or does someone just want to get your bank account details? Is your potential attacker a small-time crook or are they sponsored by a nation state or massive cartel?)
  • How long will your data remain valuable -- will the threat go away after time?
  • What have you got to lose? (Your cool? Money? Employment? People's lives? A war?)

If the stakes are unacceptably high, by all means go to extremes: erase data and destroy the medium; however, in most cases, plan to make any attack offputtingly difficult. Again, you have a choice: do you make it obvious to an attacker that it's not worth their bother (with a warning label or obvious damage), or do you let them try until they give up in frustration?

Examples:

1) You're handing your laptop down to your child for homework, having bought a shiny new one. They are not allowed to take it out of the house, and the battery doesn't hold charge anyway. You're essentially still in control of the computer. If your operating system doesn't restrict you from doing so, create a fresh user account for your child. Move/delete your old files and empty your wastebasket. You want to hang onto an Administrator or root login so you can do maintenance, but if you can, delete your old user account from that computer.

2) You want to give your old home PC to charity. The charity is unlikely to attack, but you don't know where your old computer will end up. People poke around for fun sometimes. You've got plenty of time to make backups of the data you want to keep and then do a fresh operating system install, ensuring you get it to reformat the hard disk the slow way, overwriting each sector to check for bad blocks. (If your chosen OS doesn't allow this, you can always download a Linux liveCD, boot into that and use the dd command to copy data from /dev/urandom to the machine's primary disk, usually /dev/sda. This may take many hours. Be really sure that no other disks are attached whose data you want to keep, just in case you overwrite the wrong disk! Finally, install an OS so the machine will work.

3) You need rid of that data fast! Ruthless baddies are at the door and lives are at stake. Grab a hammer or electric drill and destroy that disk! Put as many holes and dents into those shiny platters as you can. Hard disk heads fly incredibly close to the surface of the disk, with very finicky accuracy. A head crash can be caused by a particle as small as a particle of smoke getting between the head and the platter while it spins. You are filling the drive with broken particles, deforming the mirror-like disk surface, pitting it, smearing magnetic material about, and likely shattering the platters if they are glass, as some are. Trying to mount this mess on a rig in a clean-room to get data off it is only going to damage the rig and contaminate the clean-room. Any remaining options for recovering random chunks of data from fragments are going to be preposterously expensive, impractical and reliant on luck, so you should have time to leg-it and take mitigating action just in case. (You're still going to be in trouble for not encrypting life-or-death data in the first place, though.)

I haven't covered SSDs - I don't know enough about them.

1

Just burn them. Use a 20 liter steel drum, with ~0.5 cm holes punched circumferentially, just above the bottom rim. Dry oak, accelerated occasionally with paraffin, works very well. Only steel parts will remain.

mirimir
  • 726
  • 4
  • 11