How completely wipe information of overwritten files and folder?

Question

I have only a basic understanding of how files are written in a hard disk. Here I assume the files are overwritten this way.

1. File A is deleted

2. Its address is removed and showed vacant

3. New File B is overwritten above the space of File A

I am assuming here that even though we are deleting File A and overwriting with File B, we can still recover File A with its residual information using powerful software or instruments.

I have an idea to remove this residual information. Lets assume we have an imaginary hard disk with 10 GB space in which 5GB is filled with files and the other 5GB is empty. Then we select all the 5GB files in the hard disk then Ctrl+X (cut) it and then Ctrl+V (paste) the same files in same hard disk. Performing this operation for n number of steps thus overwriting it again and again. Will this completely wipe the residual information?

Answer 1

You have to stop thinking about this on the file level. For a storage device, all that matters is the sector. If one sector on a hard drive* is overwritten, the data in it is gone for good. There is no known way to retrieve it even with "powerful software", and there is no need to overwrite the same sector multiple times. Modern hard drives encode data in such a dense and complex format that a single overwrite will invariably make that data irretrievable (we can't even recover data from an old fashioned low-density audio cassette tape!). However, whether or not filling up a bunch of free space on a hard drive will actually overwrite the sensitive sectors is another matter. Due to features such as damaged sector relocation, and due to the behavior of the specific filesystem, it is not possible to guarantee that free space has been overwritten without overwriting the entire drive.

Filesystems are incredibly complex, and they are far more than flat databases of files. Cutting and pasting a file for example does not do anything but move it, which involves changing only a few bytes in filesystem metadata, regardless of how large the file is. So what about deleting a file and then filling the free space with dummy data until the drive is full? That might work, but it might not. Many filesystems contain redundant copies of information. The filesystem ext4 for example can keep copies of small files in its journal, which does not get overwritten when you wipe free space.

The exact way of storing new files is also more complex than what you have mentioned. When you delete a file, you are deleting a reference to the file (as you have surmised). However, creating a new file does not guarantee that you will be overwriting the sectors that made up the previous file. A new file will likely be strategically placed at an address on the hard drive that minimizes access latency, or which decreases fragmentation. The file may not even be deleted, but simply hidden, in order to make undeletion and incremental backups ("snapshots") possible. A new file will not simply be stuffed into the newly unallocated space in all but the very simplest of filesystems.

_{* When I say hard drive, I mean a real spinning rust. Solid state and hybrid drives work differently such that, even if you overwrite the same sector twice, the physical location that the data is saved to may be different each time.}

---

If you actually need to remove a file such that no one can recover it, you will not have many options at your disposal that preserve any other existing data on the drive. But you aren't out of luck:

1. If you have encrypted the file or drive, you can simply throw away the encryption key.

2. You can erase the entire drive, for example by using ATA Secure Erase.

3. You can destroy the drive from the outside using an expensive degaussing machine.

Deleting the file and then filling up free space, or shredding the file using data erasure tools (which typically get a list of the sectors that the file occupies, and then overwrites those specific sectors) will generally destroy the majority of the file, but comes with a high risk of incomplete erasure, with both the file's metadata and potentially small portions of the file remaining elsewhere on the drive.

Answer 2

The answer is not easy, and it depends on who you are, or rather what the threat model is.

So what is the problem at hand?

Are you an average user about to throw the old computer away, and you are afraid someone might take the harddisk, and they might read your 2007 tax report, or your 5-year old's nude-on-beach photos might land on a pedophile website?

Are you a pedophile and the police is standing in front of your door? Do you illegally sell firearms or drugs and you hear loud thumps on your door?

Are you the keeper of your nation's nuclear weapon keys? Or the member support person the Islamic State, and you are afraid your member list might be found?

Or, are you just a bit paranoid, and you want to be on the safe side, just in case?

And then of course, the question is, are you protecting against real threats, unlikely or theoretical threats, or insane threats.

And the theory?

On a typical Windows computer, the theory is that files can be un-deleted very easily, basically trivially, and a large number of tools exists to do that job. In practice, whenever I have attempted to un-delete a file which was deleted maybe two or three weeks ago, the attempt was factually unsuccessful, with only fragments being restored or the file being subtly corrupted (just enough so it won't open), and it was easier to just re-generate the document from scratch than waste time on recovering and then work with the fragmentary rubbish.
That, however, doesn't mean that someone going after your super secret supersecrets isn't possibly doing it!

On a typical non-Windows computer (say, Linux), the theory is that un-deleting files is not something that's normally expected to work at all (though there exist exceptions).

In both cases, unless some special conditions apply, data will remain on the disk and can, at least in theory, and at least partially, be recovered. And, at least in theory, it's dead simple to do.

Theory has it that once sectors on a harddrive (or solid state disk) are overwritten, their original contents are gone. In practice, you have sector reallocation on harddisks and sectors are only ever erased in large blocks on SSDs. So, what's gone is not necessarily gone at all, and what's overwritten is not necessarily (not even likely!) overwritten. That happens inside the disk without you being able to control any aspect of it.

Theory has it that on a magnetic medium you can very well recover data even after it has been overwritten (even several times). Practice has it that this was easily (and sometimes routinely) done a decade or so ago, but it's nowhere near as simple or reliable nowadays, as disk technology has evolved to a point where this becomes a really tough challenge, practically.

Now it depends on what amount of paranoia you have, or how many years of prison for you depend on data being unrecoverable. If in doubt, opening the drive and applying a neodym magnet, followed by a hammer, followed by fire is still the safest method.

But... but... That doesn't help, what do I do???

Overwriting disk contents by copying around files as you suggested is not secure, but may be a perfectly valid approach for the normal average home user. It takes no special tools, no expertise, and it has a fair chance of overwriting most of the vital data, including e.g. MFT contents change journals. Which is much better than e.g. generating a single huge file with random bytes that fills the entire free disk space (but possibly leaves file names and directory structore recoverable).

A small program which generates a hundred thousand small files filled with random data, and deletes them again afterwards is like two dozen lines of code. It's what I run on my wife's work laptop every 2 years when she's handing it in to be replaced by a new model (just in case there might be a nosey IT guy who likes to sniff around in stuff that isn't his business).
In my opinion, this (or simply copying around a few thousand ordinary files) is way enough for 99% of all people who aren't hard criminals or evemies of the state. When disposing the harddisk (no data to be retained), well, use the low-level format or hammer method (see below). While there is probably no advantage, it also doesn't hurt.

Using a typical "secure erase" tool is utterly useless because the tool has zero control over what is overwritten (reallocation and wear-levelling is invisible and uncontrollable). In fact, these tools are dangerous insofar as they give the unaware user a false sense of security (much like antivirus programs).

Using a low-level format ("factory reset", "secure erase"), presumably via the disk vendor's supplied tool, or another low-level tool will reliably destroy data on modern drives by erasing the encryption key. Virtually all modern drives use encryption to scramble data prior to writing to MLC or disk. That's not done for security purposes, but for better wear levelling. However, it also allows "secure erase" for free by simply tossing the key. The obvious downside to that is that not only part of the disk is erased, but everything.

Answer 3

As an addition to other answers, I'd like to mention some quirks, such as bad sectors, over-provisioning and temporary caches.

TL;DR: use full drive encryption

SSD overwriting

As you delete files, an SSD has to slowly erase the memory block and only then rerecord it. Instead, SSDs mark the overwritten block as needing erase and copy its data, sans deleted file, to a new block and map it to replace the overwritten one.

This means that if the file was modified or not written in one go, multiple copies of it now exist on SSD flash space until the SSD manages to erase them.

What's more is that it also happens if any file or directory that shares 64kB block with the secret file is modified in any way including just storing last access time.

The SSD should in theory schedule erasure of block that is unmapped (unused/invisible) and not yet erased but in practice unerased data may linger for some extended time.

OS and software, including file-erase software, have no access to this low-level storage space, and in fact are unable to determine if such a scheme is in effect, let alone read or overwrite any "hidden" data. However with forensic tools it may be possible to access it.

A TRIM command is not guaranteed to help as it only marks blocks as needed to be erased (as opposed to being "undeletable" in theory), not forces to erase them. It still is a good idea to enable TRIM if possible as a precaution.

SSD over-provisioning

If a drive is near-full when data is deleted or overwritten, the SSD would need to actually erase data before allowing it to be used again, which degrades performance significantly. Instead an SSD has more capacity than it shows to the OS, and uses that while data is still being erased.

For example, drive vendor may decide that for some reason it is best not to erase data in over-provisioned space unless free (erased) space is running low. In this case some data may linger if the drive is not overwritten fast enough. In practice this normally shouldn't happen because of wear leveling though.

What this means is even if you manage to overwrite all 240GB of your drive with 240GB of data, extra 16GB may still contain some of the data, and you have no idea when will it be erased, if at all.

Bad blocks

Both HDDs and SSDs may set aside parts of storage space that when read report errors. Nanometer-sized flaws on hard drive platters and flash memory transistors are inevitable and can not be diagnosed in-factory, so drives monitor errors and when they find some they decide to use blocks from a set-aside buffer instead of the "bad" ones. There is a tiny possibility that a part of your file may end up in such a "bad" block and not touched ever since. Forensic experts may recover such data irregardless. It is possible for a very frequently accessed file to end up this way if a power glitch occurs as it is being read or written to.

Drive erase methods

ATA secure erase command should in theory wipe those hidden areas, however that is not explicitly required and some drives would not actually do that.

Device manufacturers' low-level formatting utilities or commands should be more reliable, but often require advanced knowledge and skill and sometimes hardware such as an UART dongle to operate. Such utilities may not be available for every storage device and many have relevant functions disabled for customer (non-authorized professional) use, although unauthorized "leaked" versions are often possible to find.

Regardless, both methods attempt to erase all information on the device, including non-deleted files, programs, OS, and everything else. There are no tools that would actually securely erase a single file.

File encryption

Encrypting each file individually and then destroying a password should be the way to go. However if the password or code or its precursor is actually also stored on a storage device, instead of you typing it every time the file is accessed, it is also subject to a possibility of ending up stored in a hidden space.

A file password may be only derived from master password which is never written to disk and only stored in memory. This is how some (almost all?) encrypted filesystems work.

Please note that decrypted file may not be temporarily stored on the storage medium like most compression programs (7zip, winrar) do - otherwise it defeats the whole purpose. Only real way to avoid it would be to use an encrypted filesystem, such as truecrypt/veracrypt or NTFS-encryption described above (although I'm not sure if the latter one is foolproof).

Some software, such as MS Word, may also store some parts of your file in temporary files, also defeating the point if the temp directories are not encryped themselves.

Windows explorer may cache previews also possibly defeating it. There may be more leak opportunities depending on software installed such as translation software, smartphone typing prediction, screen readers, etc - even if the software is not instructed to operate on secret files, it may still analyze or cache parts of them.

It should go without saying that any malware or spyware also presents a risk even if offline. So may badly-written corporate "anti-slacking" (worker-spying) software or similar.

Thoughts on mitigation

A hardware-encrypted drive (such as "secure"-labeled USB flash drives) may help mitigate some of the issues, however some of those drives were found to not actually encrypt the data securely enough.

Some OS such as Linux and some "Live-DVDs" may allow to have the majority of OS on a read-only filesystem, eliminating threat of its "contamination" and necessity of OS encryption, however it always causes much headache because you cannot install new software or update software to secure against new threats. Yet it may be a viable solution when military-grade secrecy is required.

Conclusion

In the end it looks like a full drive encryption, including the OS, is the only sure-fire way to go for an ordinary home/office user.

Answer 4

If you are using Windows, you can use cipher, a built-in command line tool for performing encryption, which also happens to be able to remove deleted data securely.

To overwrite deleted data, one can use the /w switch
...
cipher /w:driveletter:\foldername
...
Due to the very nature of the tool, you are safe using it to securely delete data, as it will never overwrite your active files; it will only overwrite data which has been deleted by you.