151

Please note: The answers and comments to this question contains content from another, similar question that has received a lot of attention from outside media but turned out to be hoax question in some kind of viral marketing scheme. As we don't allow ServerFault to be abused in such a way, the original question has been deleted and the answers merged with this question.

Here's a an entertaining tragedy. This morning I was doing a bit of maintenance on my production server, when I mistakenly executed the following command:

sudo rm -rf --no-preserve-root /mnt/hetznerbackup /

I didn't spot the last space before / and a few seconds later, when warnings was flooding my command line, I realised that I had just hit the self-destruct button. Here's a bit of what burned into my eyes:

rm: cannot remove `/mnt/hetznerbackup': Is a directory
rm: cannot remove `/sys/fs/ecryptfs/version': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/inode_readahead_blks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_max_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/delayed_allocation_blocks': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/max_writeback_mb_bump': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stream_req': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_min_to_scan': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/mb_stats': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/trigger_fs_error': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/session_write_kbytes': Operation not permitted
rm: cannot remove `/sys/fs/ext4/md2/lifetime_write_kbytes': Operation not permitted
# and so on..

I stopped the task and was relieved when I discovered that the production service was still running. Sadly, the server no longer accept my public key or password for any user via SSH.

How would you move forward from here? I'll swim an ocean of barbed wire to get that SSH-access back.

The server is running Ubuntu-12.04 and hosted at Hetzner.

Martin Schröder
  • 315
  • 1
  • 5
  • 24
Jonas Bylov
  • 1,623
  • 3
  • 11
  • 5
  • 49
    Restore from backups. Honestly, this is one of those no-easy-way-back scenarios. – MadHatter Apr 07 '14 at 06:45
  • 322
    How do you even type `--no-preserve-root` accidentally?! :-o – ThatGraemeGuy Apr 07 '14 at 06:46
  • 151
    Greame, the keys are like right next to each other. – MadHatter Apr 07 '14 at 06:47
  • 40
    Tuesday work: Look for new job ;) Take it as a lesson why backups are needed. – TomTom Apr 07 '14 at 07:00
  • 2
    Reach for your backups and DR plan - you have both don't you ? – user9517 Apr 07 '14 at 07:20
  • 5
    Haha, thanks for all the lovely encouragement. Thankfully backups are safe. – Jonas Bylov Apr 07 '14 at 09:05
  • @TomTom He never said he had no backups; quite to the contrary. – DBedrenko Apr 07 '14 at 09:40
  • 1
    Thanks. Nice to see someone with enough brain to make backups - too many people trust in that (guild of that myself). Gratulations. Now you have the bad work of rescuing the system - but at least you ahve data. – TomTom Apr 07 '14 at 09:44
  • 2
    @TomTom this is actually **not** why backups are needed. Because, "distilled stupidity" [1] has the potential to ruin backups just the same. Backups are there for cases of accident or emergency. _([1] re-using with permission_) – sehe Apr 07 '14 at 10:06
  • 3
    @user22394: Is what we have here not an accident? Like when I almost cut off my thumb was an accident out of stupidity, but still an accident? – phresnel Apr 07 '14 at 10:40
  • 9
    an entertaining (and eye opening) read, if you don't have better tools at hand than the existing system and a few backups: http://www.ee.ryerson.ca/~elf/hack/recovery.html – Olivier Dulac Apr 07 '14 at 11:08
  • 1
    @phresnel In this analogy "--no-preserve-root" would be like a protective cover that automatically reverts back into protective position right after _every single maintenance operation do when operating the saw, or after n seconds idle_. In that case, yes it's really surprising that --no-preserve-root could be in effect. Unless, of course, it's hard coded somehwere. And that's stupidity, indeed – sehe Apr 07 '14 at 12:13
  • 46
    This sure seems like trolling to me. You can't accidentally type --i-really-mean-delete-my-whole-root. – psusi Apr 08 '14 at 01:35
  • 3
    I guess you referred [this](http://uncyclopedia.wikia.com/wiki/Rm) – iceman Apr 08 '14 at 08:21
  • 1
    :-(( Ouch! On the bright side, you're truly an Unix-administrator now, since this is a rite of passage we all (have to?) go through... of course bonus-points are given for having taken back-ups. – Baard Kopperud Apr 08 '14 at 10:38
  • 2
    First off what the hell were you doing deleting stuff on a production server, while the service is running? Why do you even have access to it? This should have been done on a staging server, tested, and then pushed onto the production server. Also if you mounted the volume containing the backups (assuming NFS share here?), why are you even working at all on the production server? Shouldn't you have been on a workstation with the backups share mounted there instead? –  Apr 08 '14 at 05:05
  • Second, as many have said, backups backups backups. A full backup each month and incremental each day (or even every hour in the case of high transaction volume) is usually the safest bet. If you had a staging server, you wouldn't even have needed them anyways. And on the off chance something awful and unexpected happened on the production deployment but everything worked on the staging server, you could have just imaged it and copied it over in about 15 minutes. –  Apr 08 '14 at 05:05
  • @Baard: yeah, but I did it when I was 11, on an IBM PS/2 running Red Hat 4. – nomen Apr 09 '14 at 04:52
  • 1
    Always use `-v` or `alias rm='rm -v'`, so you can have more chances to press `Ctrl-C` quickly enough if something is going wrong on the screen. – kenorb Apr 22 '15 at 19:15
  • 2
    If you don't have backups what are you even trying to recover? Everything is gone. Data recovery tools are your best bet but I wouldn't count on them. – André Borie Apr 10 '16 at 22:29
  • 14
    If you really don't have any backups I am sorry to say but you just nuked your entire company. – André Borie Apr 10 '16 at 22:42
  • 9
    I feel sorry to say that your company is now essentially dead. You might have an extremely slim chance to recover from this if you turn off **everything right now** and hand your disks over to a reputable data recovery company. This will be extremely expensive and still extremely unlikely to really rescue you, and it will take a lot of time. – Sven Apr 10 '16 at 23:07
  • 6
    Backups need to be offsite, offline, and incremental. That you could delete them from your main server means they weren't what I would call backups. – Tim Apr 10 '16 at 23:09
  • 18
    You're going out of business. You don't need technical advice, you need to call your lawyer. – Michael Hampton Apr 10 '16 at 23:10
  • 5
    I'm curious about how the command succeeded though - a simple `rm -rf /` should fail (or at least it does fail on my personal server) unless the `--no-preserve-root` option is provided. – André Borie Apr 11 '16 at 03:25
  • 2
    Could we answer instead with a reference checklist of how this should have been prevented? – Houston Fortney Apr 11 '16 at 03:38
  • 1
    1) Tape backup 2) Dropbox synchronization script 3) Manually copy your incremental backups to a hard drive 4) BitTorrent Sync etc – Tim Apr 11 '16 at 06:49
  • 2
    He must be trolling , can't rm -rf / without --no-preserve-root – Phyo Arkar Lwin Apr 11 '16 at 09:56
  • And if there is --no-preserve-root , you must just be doing that on purpose. – Phyo Arkar Lwin Apr 11 '16 at 10:04
  • @PhyoArkarLwin You can do `rm -rf /*`. Or you could be running an old OS; `--no-preserve-root` wasn't brought into GNU rm until 2006, IIRC. – Jenny D Apr 11 '16 at 10:50
  • Take look at this link : https://unix.stackexchange.com/questions/80270/unix-linux-undelete-recover-deleted-files My2cents –  Apr 11 '16 at 11:20
  • 2
    I don't know if this helps, but this story is entertaining as well. In this story, "rm -rf /" was allowed to run accidentally for a bit before being interrupted. The author tells of some very creative engineering using tools in unusual ways to reconstruct "/etc" and then "/etc/passwd", etc. [Unix Recovery Legend](http://www.ee.ryerson.ca/~elf/hack/recovery.html) – Kevin Wright Apr 08 '14 at 18:34
  • Next time, in Bash: set -o nounset – Luchostein Apr 17 '16 at 13:15

9 Answers9

224

Fact is? At this point, there's no simple/easy automatic fix for this. Data recovery is a science and even the basic, common tools need someone to sit down and ensure the data is there. If you're expecting to recover from this without massive amounts of downtime, you're going to be disappointed.

I'd suggest using testdisk or some file system specific recovery tool. Try one system, see if it works, and so on. There's no real way to automate the process but you can probably carefully do it in batches.

That said, there is a few very scary things in the questions and comments that ought to be part of your after action reports.

Firstly, you ran the command everywhere without checking it first. Run a command on one box. Then a few, then more. Basically if something goes wrong, its better to have it affect a few rather than all your systems.

Secondly

@Tim how to do a backup without mounting a remote drive on the server?

Scares me. File level one way backups are a solved problem. Rsync can be used to preserve permissions and copy over files one way to a backup site. Accidentally something? Reinstall (preferably automatically) rsync back, and things work. In future, you might use file system level snapshots with btrfs or zfs snapshots and shipping those for system level backups. I'd actually toy with separating application servers, databases and storage and introduce the principle of least privilege so you would split up the risk of something like this..

I know there is anything I can do. I now need to think how to protect myself

After something has happened is the worst time to consider this.

What can we learn from this?

  1. Backups save data. Possibly careers.
  2. If you have a tool and arn't aware if what it can do, its dangerous. A jedi can do amazing things with a lightsaber. A roomful of chimps with lightsabers... would get messy.

  3. Never run a command everywhere at once. Seperate out test and production machines, and preferably do production machines in stages. Its better to fix 1 or 10 machines rather than 100 or 1000.

  4. Double and triple check commands. There's no shame in asking a co worker to double check "hey, I'm about to dd a drive, could you sanity check this so I don't end up wiping a drive?". A wrapper might help as well, but nothing beats a less tired set of eyes.

What can you do now? Get an email out to customers. Let them know there's downtime and there's catastrophic failures. Talk to your higher ups, legal, sales and such and see how you can mitigate the damage. Start planning for recovery, and if needed you're going to have to, at best, hire extra hands. At worst, plan for spending a lot of money on recovery. At this stage, you're going to work at mitigating the fall out as well as technical fixes.

Journeyman Geek
  • 6,969
  • 3
  • 31
  • 49
  • My backups are 95% ZFS snapshots and 5% rsync (to a ZFS dataset that is then backed up!) for the few things I can't put directly on ZFS, like Linux root filesystems. Restoring is dead simple, albeit bandwidth intensive if I need to go to the off-site backup. – Michael Hampton Apr 11 '16 at 08:12
  • 9
    @MarcoMarsala If you mounted anything before using rsync, you weren't doing it correctly. You should be using rsync over ssh. – Michael Hampton Apr 11 '16 at 08:19
  • 69
    I'd add to this excellent answer: Step away from the computer. Do not try to fix anything until you've calmed down. You're already looking at some serious downtime; taking the time to think things through instead of wrecking your systems even more (as in the `dd` issue above) is not going to make it worse. – Jenny D Apr 11 '16 at 08:44
  • 2
    My answer covers that. The crucial mistake is in running a untested command across the entire server farm – Journeyman Geek Apr 11 '16 at 09:08
  • 22
    Any idea why the command actually ran? If `$foo` and `$bar` were both undefined, `rm -rf /` should have errored out with the `--no-preserve-root` message. The only way I can think of that this would have actually worked on a CentOS7 machine is if `$bar` evaluated to `*`, so what was run was `rm -rf /*`. – terdon Apr 11 '16 at 09:23
  • 8
    Everything said apart, one more simple thing - Do not multi-task as a human. Do not mix backup or copy operations with the change operations in the same script. Take a backup (and verify) *before* you change anything. Start change operations *only after* your backup/copy operations are complete and safely disconnected away. (Your rm changes the state and hence is a change operation). Holds true not only for data-center severs, but things as simple as editing a plain text document as well. Also, periodically verify backup restore/recovery process as well. – Abhitalks Apr 11 '16 at 09:39
  • 2
    Yeah, quite a lot of the underlying problems are process issues ('soft' skills) over technical ones ('hard' skills). – Journeyman Geek Apr 11 '16 at 10:28
  • 1
    The "verify the backup" thing that @Abhitalks mentions is more important than it may sound: http://news.softpedia.com/news/server-snafu-makes-microsoft-beg-for-ca-audit-data-from-its-partners-501357.shtml – Rhymoid Apr 11 '16 at 19:21
  • 9
    I love the stylism in "Accidentally something?". That must mean the word "removed" was "deleted" or "dropped" accidentally. – sehe Apr 11 '16 at 20:17
  • 4
    Good comments. I would just add that all my shell scripts have `set -u;set -e`. The posted scenerio would not have occured if he had `set -u`. – Aaron Apr 12 '16 at 15:27
  • @Aaron what does this command do? – Veve Apr 14 '16 at 14:46
  • 20
    @MarcoMarsala well at least you're famous now http://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html – Martin Smith Apr 14 '16 at 15:06
  • 9
    `set -u` will cause bash to exit if there are undefined variables. For example, if `foo` and/or `bar` are not defined, then `rm -rf ${foo}/${bar}` would cause the shell script to exit non-zero status. – Aaron Apr 14 '16 at 15:46
  • @JourneymanGeek I think you mean "art", not "science". Or perhaps you mean "is not a science". See https://en.wikiquote.org/wiki/Exact_science – daveloyall Apr 14 '16 at 16:04
  • 2
    Testing should never, ever, ever be done in production. If you haven't already run something, it should be run in a test environment first. That can help avoid explosions like this in production. – erik258 Apr 14 '16 at 16:37
  • @Aaron You can just use `set -eu`, no need to two `set` commands ;-) I also like the `-C` option by the way ("noclobber") which prevents overwriting existing files with `>` (you can still use `>|` to force this though). – Martin Tournoij Apr 14 '16 at 20:52
  • 1
    Slashdotted - https://developers.slashdot.org/story/16/04/14/1542246/man-deletes-his-entire-company-with-one-line-of-bad-code – DavidPostill Apr 14 '16 at 21:01
  • 2
    Congrats, this Q&A [made the local paper](http://www.sunshinecoastdaily.com.au/news/man-accidentally-deletes-his-entire-company/2997367/). – aroth Apr 15 '16 at 02:18
  • 1
    @bleemboy Good to see you got your data back. Goes to show how shadowy the world of data recovery still is even for experts in ServerFault. On a HDD platter, data is offset allowing the same bit to be overwritten 7 times but still read. You can't use a HDD internal read head, but need a better one to read the data back out. Being uncommon, it seems like magic. I don't think Michael's comment on the question was very useful. – Kind Contributor Apr 15 '16 at 04:14
  • 3
    "On a HDD platter, data is offset allowing the same bit to be overwritten 7 times but still read." Citation needed – Journeyman Geek Apr 15 '16 at 04:30
  • @JourneymanGeek there is a software that does tens of passes to wipe out hard drive. Why would anybody implement that if data is so fragile anyway? – Vanuan Apr 15 '16 at 05:23
  • 2
    Actually, the 'guttmann' method has different patterns for different encoding, and was designed for an era when magnetic domains are larger. With modern drives as little as a single pass would probably be enough, and a few passes with random data ought to be safe. – Journeyman Geek Apr 15 '16 at 05:27
  • @JourneymanGeek Good point. I don't have the reference - I read it once somewhere. Higher densities with smaller magnetic domains could make my assertion a relic of the past. However there still are limitations with mass production versus forensic tools. A commercial HDD could be like an optical lab microscope and forensic tool like an Electron Tunneling Microscope. Certainly the gap would be closing, but don't discount advances on the forensic tooling side equally. Also, I concede that SSDs are quite pervasive these days, making my assertion obsolete. Just as well rm doesn't block erase. – Kind Contributor Apr 15 '16 at 05:43
  • I've no evidence of data recovery using a electron tunneling microscope ever working. Most forensics uses pretty standard tools, with lots of precautions. Outside of a write blocker this sort of forensics is entirely software based. – Journeyman Geek Apr 15 '16 at 05:45
  • @JourneymanGeek That was an analogy: Nor are hard disks read with optical microscopes. Here's a good answer against my assertion: http://security.stackexchange.com/a/10474. My point may have been urban legend. – Kind Contributor Apr 15 '16 at 05:47
  • 2
    I majored in computer forensics, I vaguely know what I'm talking about, and a vague, cargo cultish belief in guttman's paper is a pet peeve. Its a *great* paper, but its written for hardware of a very different era. – Journeyman Geek Apr 15 '16 at 06:32
  • 1
    Is this [funds raise](https://www.generosity.com/emergencies-fundraising/help-to-pay-a-data-recovery-service) for real? – DanielV Apr 15 '16 at 09:06
  • @DanielV fundraiser? what fundraiser? – tombull89 Apr 15 '16 at 10:19
  • @tombull89 yeah, follow the link – DanielV Apr 15 '16 at 10:31
  • @daniel it appears to have been taken down. – Martin Smith Apr 15 '16 at 10:47
  • Just say cheese Marco Massala. You are on all internet news. The guy who deleted all his company with one command line. Kids of the future reading this: DO BACKUPS no matter how simple are the tasks you are doing –  Apr 15 '16 at 11:59
  • 1
    And, we are still curious how your `rm -fr /` achieved your company deletion WITHOUT the `--no-preserve-root` –  Apr 15 '16 at 12:09
  • 1
    @nwildner The magic of CentOS I guess? – Xerz Apr 15 '16 at 12:12
97

Boot into the rescue system provided by Hetzner and check what damage you have done.
Transfer out any files to a safe location and redeploy the server afterwards.

I'm afraid that is the best solution in your case.

faker
  • 17,326
  • 2
  • 60
  • 69
93

When you delete stuff with rm -rf --no-preserve-root, its nigh impossible to recover. It's very likely you've lost all the important files.

As @faker said in his answer, the best course of action is to transfer the files to a safe location and redeploy the server afterwards.

To avoid similar situations in future, I'd suggest you:

  • Take backups weekly, or at least fortnightly. This would help you in getting the affected service back up with the least possible MTTR.

  • Don't work as root when not needed. And always think twice before doing anything. I'd suggest you also install safe-rm.

  • Don't type options that you don't intend to invoke, such as --no-preserve-root or --permission-to-kill-kittens-explicitly-granted, for that matter.

Amal Murali
  • 1,032
  • 7
  • 10
  • 22
    Similarly, unless you REALLY MEAN IT, don't add the `--please-destroy-my-drive` parameter to `hdparm`. – MikeyB Apr 08 '14 at 06:17
  • 3
    I'd like to add; "Triple-check your arguments (and options) when working as root", "Check your CurrentWorkingDirectory (before doing something like rm -rf *)", and "Use full-paths to commands (don't relay on $PATH). – Baard Kopperud Apr 08 '14 at 10:41
48

I've had the same issue but just testing with a harddrive, I've lost everything. I don't know if it'll be useful but don't install anything, don't overwrite your data, you need to mount your hard drives and launch some forensics tools such us autopsy, photorec, Testdisk.

I strongly recommend Testdisk, with some basics command you can recover your data if you didn't overwrite it.

kasperd
  • 29,894
  • 16
  • 72
  • 122
Octo
  • 581
  • 4
  • 5
  • 9
    I would definitely recommend takign the storage offline if at all possible and re-mounting as 'read only' if you can at all. Whether with a livedisk or another server instance. – mhouston100 Apr 12 '16 at 00:21
  • 2
    I'd even consider doing a dd bitcopy of the original disk to a new disk from a read-only mount of the original disk just to be safe. – Jim Apr 14 '16 at 19:49
  • 3
    «these tools won't recover the file name and path» Yes, they do. Of the 3 mentioned tools, only one (Photorec) performs carving. – Andrea Lazzarotto Apr 16 '16 at 16:34
34

The best way to fix a problem like this is to not have it in the first place.

Do not manually enter an "rm -rf" command that has a slash in the argument list. (Putting such commands in a shell script with really good validation/sanity routines to protect you from doing something stupid is different.)

Just don't do it.
Ever. If you think you need to do it, you aren't thinking hard enough.

Instead, change your working directory to the parent of the directory from which you intend to start the removal, so that the target of the rm command does not require a slash:

cd /mnt

sudo rm -rf hetznerbackup

Community
  • 1
Monty Harder
  • 633
  • 4
  • 9
  • 33
    I always put the -rf at the end of the argument list, so `rm /bla/foo/bar -rf`. At least that way I'm not into a lot of trouble when I accendentily press return after typing the `rm /` part. – Jens Timmerman Apr 14 '14 at 16:37
  • 5
    Similarly, when removing "*~" files, I type the tilde first, then add in the asterisk. – tekknolagi Apr 24 '14 at 01:59
  • 4
    So you'd rather delete your home than everything in the current directory ?!? – greg0ire Apr 17 '16 at 13:50
  • @greg0ire No, i think he wanted to say, that within `/mnt/hetznerbackup`, he had to use "/" to mark everything inside that folder.. but from parent, only `hetznerbackup` is enough, without slashes. – T.Todua May 16 '16 at 19:41
  • 1
    @tazotodua : I was referring to tekknolagi's comment – greg0ire May 17 '16 at 14:06
  • I don't like the tradeoff of deleting home...how about typing `rm ./~` first, then change to `rm ./*~` – krubo Mar 24 '19 at 21:01
15

I would try to recover backup machine, where all copies were stored:

  • 1st step - Make a backup of this erased "backup machine" drives with dd comand.
  • 2nd step - Use testdisk to recover files.

So lets say you want to recover 1TB, You will need extra 2TB, 1TB for backup (1st step) plus 1TB for recovery (2nd step).

I did similar mistake with alias rm -fr [phone rang] and cd to precious directory. Now i always think twice and recheck couple times before i use rm or dd command.

Cristian Ciupitu
  • 6,226
  • 2
  • 41
  • 55
Abc Xyz
  • 608
  • 1
  • 8
  • 16
  • 6
    Pretty much zeroed your disk by doing that. That seriously makes it a lot harder to recover. There's a good reason the OP suggested you tried using testdisk, and recovering first, and while dd's syntax can be a little odd, that's a good reason to double and triple check before you run the command. You only wiped one server, right? – Journeyman Geek Apr 11 '16 at 07:16
  • 1
    You can still recover, depends how long you allowed `dd` to erase your last chance. – Abc Xyz Apr 11 '16 at 13:53
  • 129
    sorry to say that, but i feel huge troll in this question... – tymik Apr 11 '16 at 22:19
  • 3
    hope u feel small troll in the answer :) – Abc Xyz Apr 11 '16 at 22:27
  • 5
    To be honest. I'm not sure you're real. If you are, you're probably in the wrong job... – leftcase Apr 14 '16 at 19:05
7

As mentioned in another answer, Hetzner has a rescue system. It includes both a netboot option with ssh access as well as a java applet to give you screen and keyboard on your vserver.

If you want to recover as much as possible, reboot the server into the netboot system and then log in and download an image of the filesystem by reading from the appropriate device inode.

I think something like this should work:

ssh root@host cat /dev/sda > server.img

Of course the redirection is done by the shell before the ssh command is invoked, so server.img is a local file. If you want just the root file system and not the full disk, replace sda by sda3 assuming you are using the same image as me.

kasperd
  • 29,894
  • 16
  • 72
  • 122
  • could maybe be : `ssh root@host cat /dev/sda | gzip -c - > /path/to/dir_on_huge_partition/server.img.gz` (the on-the-fly gzip will or won't help depending on what the content of the filesystem is...) – Olivier Dulac Apr 07 '14 at 11:06
  • @OlivierDulac Using gzip that way would send the data uncompressed over the network and then compress it on the receiving side. I assume the result you intended to achieve was to compress the data while being transferred. The local image could be stored compressed or not, but tools you'd like to apply to that image later will not work with the compressed version. If all you want to achieve is compression of data while in transit, you can make use of the compression feature in ssh. It can be enabled with `-C` if it is not already enabled in your configuration. – kasperd Apr 07 '14 at 11:16
  • 2
    I was more trying to reduce the size of the file. But if you want to save bandwidth (good idea) : just add quotes: `ssh root@host "cat /dev/sda | gzip -c - " > /path/to/dir_on_huge_partition/server.img.gz` (the -c option of ssh is usually good too, but you'd still need to compress at the end, as ssh will only compress at entrance of its tunnel and uncompress before sending to stdout) – Olivier Dulac Apr 07 '14 at 11:20
3

How would you move forward from here?

I would swear off using rm for the rest of my life and think that it's madness that trash-cli isn't the default removal command on nix systems.

https://github.com/andreafrancia/trash-cli

I would make sure it is the first thing I install on a brand new system and alias rm to something that tells people to use trash-cli instead. It would also include a note about another alias that actually runs /bin/rm but tells them to avoid using it in most cases.

:( True story

chicks
  • 3,639
  • 10
  • 26
  • 36
Gerry
  • 338
  • 2
  • 3
  • 12
  • 3
    In my experience, these kind of tools are more like a nuisance than an actual help - sooner or later, and after some swearing, you'll remove it. It might be ok for a workstation, but in many if not most situations when you are doing administrative work on a server, you really need to delete the data, not just move it somewhere else (and if that were the case, just use mv instead). Besides, automatically moving data to a trash folder might lead to serious issues by itself (e.g. trash not on the same filesystem, security). – maetthu Apr 17 '16 at 10:01
  • @maetthu Oh of course things are removed after they have been in the trash for a certain number of days. Ubuntu desktop does this to items which have been in the trash more than 30 days. On a server you might want something shorter, eg. `trash-empty 5` in a cron. The point is to allow you some grace period because humans make mistakes. – Gerry Apr 17 '16 at 13:29
  • Isn't it better to have a working desaster-recovery plan instead of banning essential system tools? – user292812 Apr 17 '16 at 15:56
  • @user292812 I did not suggest banning /bin/rm, just that it shouldn't be the first option in most cases (note the /bin/rm alias). Your question also suggests a false choice between disaster recovery and a human friendly deletion option. You should have both. – Gerry Apr 17 '16 at 17:56
  • 1
    A two-step remove process can save a lot of trouble: 1. move to trash (verbosely), 2. empty trash. I alias such a script to "rm" and it has saved me from accidentally deleting important things many times. – Sam Watkins Apr 18 '16 at 13:54
  • @SamWatkins yes, and the best two-step remove process is (1) type the `rm` command, then (2) think carefully before pressing `enter`. It is a good idea to get into the iron habit of pausing for thought before doing *anything* irrevocable; it is a less good idea to get into the habit of expecting code to save you from your failure to do this. – MadHatter Jul 04 '18 at 07:35
  • @MadHatter you forgot step 3. Don't make a mistakes (or be a human who thinks they never make mistakes). – Gerry Jul 05 '18 at 06:42
  • @Gerry we all make mistakes, and saving us from those is one thing that regularly-tested backups are for. Weenie hacks/alternatives to `rm` generate a false sense of security, and people relying on them just get bitten by all the other ways to delete files eg output redirection, emacs' delete-file, etc. instead. You will **never** outfit a Unix system to save you from all possible errors you can make, so make sure you have good backups, and restore from those when you make mistakes. I do. It works fine. – MadHatter Jul 05 '18 at 09:00
  • The other thing I do (at work) to save me from myself, is commit everything constantly to a mega git repo with all my junk in it. That has saved me several times also. At home I commit stuff to individual git repos and sync out to servers around the world as a sort of reflex action like saving a file. – Sam Watkins Jul 08 '18 at 17:27
2

I would advice in such case is unmount and use debugfs, and with help of lsdel you can list all recently removed files, which where not cleaned up from journals and then dump needed files. Fast search link for the same: http://www.linuxvoodoo.com/resources/howtos/debugfs

hope it will help someone. ;)

And yes, once of suggestions is to make script, which moved ream rm to real.rm and symlinc mv to rm ;)

BiG_NoBoDy
  • 138
  • 1
  • 8