Accidentally rm -rf /usr/* as root, what now?

Question

A colleague of mine accidentally deleted /usr/* data by running: rm -rf /usr/*. And it's now a big issue. We had a lot of good data on that machine. Most of the commands are not working as a result. Is there anyway I can recover the machine? I'm not really an expert in data-recovery.

Please help me point to a solution/procedure. Some information regarding that machine follows. Do let me know if more information is required.

-bash-3.00# uname -a
Linux penguin 2.6.9-5.EL #1 Wed Jan 5 19:22:18 EST 2005 i686 i686 i386 GNU/Linux
-bash-3.00# fdisk -l

Disk /dev/hda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          13      104391   83  Linux
/dev/hda2              14        9729    78043770   8e  Linux LVM
-bash-3.00#

Answer 1

First and foremost, boot off a live CD or recovery disk and back up your data. You may want to include system configurations from /etc, too.

You can try doing a reinstall over what you have, leaving your partitions you want to keep untouched. As long as you weren't keeping your good data in any system partitions (and let's hope not under /usr), you should be able to preserve those during your install.

If all else fails, install from scratch and restore from the backup.

Consider changing your colleague's access.

Answer 2

My recommendations:

1. Google "undelete for Linux", and you'll find something such as http://www.r-tt.com/data_recovery_linux/. You can use this to undelete any documents that you want to recover.
2. Take a step to avoid this in the future. What you want is a method to make a copy of the partition, something like Acronis TrueImage for Linux. If you run Acronis TrueImage for Linux, it copies a complete clone of the hard drive to a file (which you can safe offline for safety), so you can always restore from the bare metal if anything goes wrong in the future.

Answer 3

I am presuming that you are using Redhat simply because that is what I am using:

Step 1- Since you shouldn't have any actual data store in /usr (man hier), you should be able to reinstall all the system binaries with:

cd /root
rpm -Va > filename
rpm -qf `cat filename | awk '{print $3}'` | sort | uniq > filename2
yum reinstall `cat filename2`

If you do however have some data in /usr that wasn't installed by the distro, then forget everything I just said as using the disk will lessen the chances that you will be able to recover the deleted data.

Answer 4

First rule of data recovery: Stop using the disk

Undelete is possible, as Gravitas suggests, but the more you use the disk, the greater the chance the freed sectors will be overwritten by new data. Even just log files on an idle, booted system can be enough to scupper your recovery chances. The recommendations above are good, but make sure you do them with the disk as a slave; either with a Live CD or by attaching and mounting the disk on another system.

As for procedure. If you feel this was a one-off, then consider it a lesson learned. If you're worried it might hapen again, change the root passwords and implement a tightly controlled sudo (http://aplawrence.com/Basics/sudo.html)

After that, I wouldn't trust the system until it had been reinstalled from scratch. You might get it back to a workable state, but it's safer to blitz it.

Answer 5

Undelete kludges can work if you've lost a file or two, but with such a large chunk of data removed, I don't think chasing that will be a productive use of time.

The suggestions of copying over files from another system may get you back up and going, but may leave your system in an "impure" state, where not everything is back where it belongs.

My recommendation is to do a clean install to another disk (or another computer), and then mount the old drive manually (or boot the old computer with a live CD) and copy over your important data.

Then, treat this as a warning, and

1. Make sure you have an automatic, up-to-date backup of that "good data", so you're never in this situation again.
2. Review who has administrative access to the system, and set policies for when and how it is used. Did you colleague need to be running as root to do what they were trying to do?

Answer 6

Try to install the very same distro (RHEL4, judging by the kernel) somewhere else and manually copy important binaries and libs under /usr. Try to get a list of installed packages by rpm -qa (rpm is under /bin, and lists and variable stuff is kept under /var so this may work to a some extent). Reinstalling from scratch would be a best option, but if it is a critical system this will put the fire out. Well, the weekend is coming, so it could be a busy one :)

About prevention: If you create the file called "-i" (without the quotation marks) it will prevent someone to rm -rf * in that directory by accident because it will ask for every file. Try it in a test directory. You can create the file with "touch -- -i" (double hypen stands for "do not interpret anything after that as an option".

Answer 7

You can copy the the whole /usr directory from the other same h/w and configuration server or machine. It will work. But just make sure you make the symlinks as we

Answer 8

My recommendation would be that you should make a copy of this disc/discs before trying anything like a re-install, undelete, or recovery. If you have good backups it's probably fairly safe to just boot into a rescue CD and recovery /usr from the backup, but I'm assuming that you don't have good backups if you're asking this question. :-)

If you have data you'd like to preserve on the disc, ALWAYS try your recovery on a copy of the disc, never the original.