-2

I have a linux server (Debian squeeze) set up. It's primary focus is to run LAMP. I also use it for stuff like mumble (voice chat) and minecraft servers.

In the beginning everything was running fine. Then things started to happen. Stuff like java failing to run without headerless mode, and completely losing the ability to resolve hostnames. I fixed the hostname resolving by a reboot - which I've heard isn't really a desired fix on linux systems.

I do have physical access to the server computer (I built it myself), but it's not located in my house. So I use SSH to interact with it. Yesterday I noticed that it doesn't show the path in the ssh client but rather only "-bash-4.1#".

Also after the reboot when I tried starting the mumble server it just says missing file: "/etc/mumble-server.ini"

This server contains vital (and confidential) company information. Should I be worried that some hacker got into it? Did I install some kind of malware on it?(Doubt I did)

EDIT: I cannot seem to access the website it is supposed to be hosting. It is a vital part of the companys web services. This is real bad.

I also now remember mistakenly running something like rm .* in /etc with root a couple weeks ago. It worked fine then so I ignored it. Is there anything I can do to fix this? Did I really screw up?

UPDATE: It seems apache fails to run due to a missing mime.types file. It should be located in /etc. Is there a command to restore it?

UPDATE: Since everything else failed, I am reinstalling the OS and server software. Good thing I have all the vital and confidential info backed up regularly.

Axel Latvala
  • 201
  • 1
  • 4
  • 12
  • 2
    If you remember where in /etc/ you ran that rm command it would help, based off your comments bellow and that you have very little time, a really tacked on temporary solution would be to install the same debian version on a VM, install critical packages and compare what is missing between that VM and your server ( run find /etc on both, diff files ). in etc. It would likely be enough to get apache going. – Pratik Amin Jul 25 '12 at 02:21
  • Do you have any backups? – jscott Jul 25 '12 at 02:24
  • 1
    When the machine is back up it would be a very good idea to reconsider hosting business services alongside personal ones (MC & mumble). If you ran the `rm` command the other week then isn't it likely the site has had issues since then... ? Also this seems like more than just apache has been affected due to your mumble issue above. Sounds like the `rm` was run in the /etc and not /etc/httpd so who knows what else might be broken... Backups or new build time! – Beeblebrox Jul 25 '12 at 02:32
  • Ok I got the server running again. Mime types are real messed up. Can't display css files. could anybody provide a mime.types file that works? – Axel Latvala Jul 25 '12 at 02:32
  • @foocode seems apache had the config files loaded in memory. The problems only started when I rebooted. – Axel Latvala Jul 25 '12 at 02:34
  • 1
    `history | grep rm`? – Shane Madden Jul 25 '12 at 02:34
  • @shanemadden Ok here they are: `rm .* /backup/eResk/` and `rm .* /backup/eResk/` Both were ran from /etc – Axel Latvala Jul 25 '12 at 03:11
  • Do you have mod_mime enabled for apache? 'a2enmod mime' (the module name may be slightly different) – Pratik Amin Jul 25 '12 at 04:02
  • @PratikAmin I do have mime mod enabled. I checked it. – Axel Latvala Jul 25 '12 at 04:07
  • You could try just putting this file: http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types into /etc/apache2/mime.types, however that may not fix your problems as there could be further problems – Pratik Amin Jul 25 '12 at 04:11
  • Standard response to this kind of situation: Restore from your backup. – John Gardeniers Jul 25 '12 at 08:25
  • 4
    I'm not judging, but I find the combination of "vital (and confidential) company information" and Minecraft on the same server quite interesting. – basvdlei Jul 25 '12 at 10:07
  • @basvdlei I have the minecraft server set up in such a way that it can only access files it owns. It is run by a dedicated minecraft user who has no privileges anywhere exept its own home. – Axel Latvala Jul 25 '12 at 13:53

3 Answers3

5

Most likely the cause of your problems is the rm command that you ran in /etc. I would suggest reinstalling the OS.

bahamat
  • 6,193
  • 23
  • 28
Sameer
  • 4,070
  • 2
  • 16
  • 11
  • Is there ANY other way. This system is NEEDED in 6hours – Axel Latvala Jul 25 '12 at 01:45
  • Atleast get apache up? I can connect to the mysql server. – Axel Latvala Jul 25 '12 at 01:46
  • 2
    @AxelLatvala - 6 hours should be plenty of time to get the bare OS installed and restored from backup. – EEAA Jul 25 '12 at 02:07
  • What if I told you that I can't gain access to the facility in which the server is located before it's too late? Also I didn't do any OS backups, only data (mysql server + webapp and all it needs) – Axel Latvala Jul 25 '12 at 02:13
  • UPDATE: It seems apache fails to run due to a missing mime.types file. It should be located in /etc. Is there a command to restore it? – Axel Latvala Jul 25 '12 at 02:16
  • 1
    Try running dpkg-reconfigure apache2 – Pratik Amin Jul 25 '12 at 02:19
  • 1
    If you're missing files from `/etc`, you need to restore them from your latest, functional, backup. – jscott Jul 25 '12 at 02:35
  • @AxelLatvala, Back up your httpd.conf and web data and reinstall httpd - which'll at least restore all the server-required files. Unfortunate as it is, without backups you're in an unpleasant situation - you've gotta get stuff back up without compromising client data. So make sure you have a clean & 'safe' backup of that before doing anything else. Then you're free to do whatever duct-tape jobs you need in order to get things sorted by sunrise. :) – Beeblebrox Jul 25 '12 at 02:39
  • Without an inventory we don't know what you lost with that rm commend. Re-install is the clean solution (after making backups of what you configured and your data). If you have or can buy a new hard drive, put that it and install to that. Then hook up the old hard drive as the 2nd (/dev/sdb) and copy in old data. Another option is a 2nd machine if you can get one (even a laptop). – Skaperen Jul 25 '12 at 02:56
  • @foocode I did apt-get uninstall apache2 and the apt-get install apache2 so it reinstalled. The mime types still won't work – Axel Latvala Jul 25 '12 at 02:59
4

Your best bet to figure out exactly what you did, and where, is to check your sudo logs. Here's an example entry:

Jul 24 22:38:08 node1 sudo:    scott : TTY=pts/0 ; PWD=/home/scott ; USER=root ; COMMAND=/bin/rm marker_file

So we can see that I issued the command /bin/rm marker_file, with no additional arguments, from the directory /home/scott. So what this tells us is that I deleted the file /home/scott/marker_file.

Extrapolating this out we can figure out exactly what's missing, so you have a better idea which files to pick out of your backup. For instance, if we saw the log message

Jul 12 08:38:08 node1 sudo:    scott : TTY=pts/0 ; PWD=/etc/httpd/modules ; USER=root ; COMMAND=/bin/rm -rf tmpfile *

then we can figure out that you recursively deleted everything starting in the directory /etc/httpd/modules. Since this gives us both the timestamp of the event, as well as the list of affected files, we can pretty easily figure out what needs to be restored.

Additionally, I would make the guess that you were attempting to tab-complete the group of files that started with tmpfile without realizing that only one existed. I jumped to this conclusion based upon the fact that, generally, bash will add a space to the end of a fully matched file when tab-completing. That would be a complete conjecture, however, and only worth mentioning as a potential explanation of the incident.

Scott Pack
  • 14,717
  • 10
  • 51
  • 83
  • I pretty much do remember the command i ran. It was (in /etc). `rm *.* /backups/` – Axel Latvala Jul 25 '12 at 03:02
  • @AxelLatvala - I refer you to Shane Madden's comment on the question: `history | grep rm` - this will tell you exactly what you ran. – Mark Henderson Jul 25 '12 at 03:04
  • Ok here they are: `rm .* /backup/eResk/` and `rm .* /backup/eResk/` Both were ran from /etc – Axel Latvala Jul 25 '12 at 03:08
  • @AxelLatvala: That's actually not too terrible, since I wouldn't expect many dotfiles in `/etc`. If you got that from history, remember that by default it replaces your history file every time you close your session. So if you have multiple login sessions open at once, then data loss will occur. – Scott Pack Jul 25 '12 at 03:20
  • @scottpack Ok. Thats good. Now I need to get my apache server to send right mime types. Any idea? – Axel Latvala Jul 25 '12 at 03:22
  • @AxelLatvala: I think you missed my point. Based on your description, your system is clearly hosed and needs to be rebuilt from a known good backup. The commands you listed don't look terrible enough to be the likely culprits. – Scott Pack Jul 25 '12 at 03:43
1

Undoing (most of your) mistakes in Debian, such as deleting vital configuration files from /etc, can be undone by reinstalling all packages. This can be done with the following command:

aptitude reinstall '~i'
basvdlei
  • 1,296
  • 8
  • 13