0

One of my user accounts on an Ubuntu 10.04.3 server was hacked, and I'm not sure how. The password was strong. A cronjob was installed in my user's crontab running and executable in /var/tmp/.aw

The /var/tmp/.aw directory contained a collection of executables including one called bash.

I've examined my ~/.bash_history and found some very suspicious stuff. I provide the relevant snippets below.

    w
    ls
    passwd
    cd /var/tmp
    w
    ls
    wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
    w
    wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
    w
    cat /prooc/cpuinfo
    cat /proc/cpuinfo
    exotr
    wq
    w
    exit
    w
    ls
    passwd
    cd /var/tmp
    ls
    wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
    ls
    tar xzvf IPmech.tgz
    rm -rf IPmech.tgz
    cd .aw
    s
    ls
    ./autorun
    chmod +x *
    ./autorun
    ./start TKLL
    ls
    rm -rf m.ses
    ps x
    kill -9 4350
    ls
    ps x
    rm -rf m.ses
    kill -9  4460
    ls
    ps x
    w
    ls
    nano 192.168.0.100.user2
    rm -rf *seeN8
    ls
    rm -rf *see*
    ls
    nano m.set
    rm -rf m.ses
    ps x
    kill -9 4582
    ls
    ps x
    kill -9 4645
    rm -rf m.ses
    ls
    ps x
    kill -9 4693
    ls
    rm -rf m.ses
    ps x
    kill -9 4733
    rm -rf m.ses
    ps x
    kill -9 4757
    ls
    nano m.set
    rm -rf m.ses
    ps x
    kill -9 4800
    we
    w
    ls
    ps x
    kill -9 4878
    ls
    rm -rf m.ses
    ps x
    kill -9 4926
    ls
    w
    ps x
    ls
    kill -9 4964
    w
    exit
    w
    ls
    ps x
    cd /var/tmp
    w
    ls
    exit
    sudo su
    passwd
    ls
    ls -al
    ls .ssh/
    rm id_dsa.pub 
    touch .sudo_as_admin_successful 
    sudo su
    passwd
    it is
    sudo su
    w
    echo "yay :D" > /dev/pts/9
    echo "I take it it's working..." > /dev/pts/9
    w
    echo "Is this annoying???" > /dev/pts/9
    w
    exit

Specific questions:

  1. What is exotr? I can't locate or find it with which, nor is there man entry for it
  2. What is IPmech, a google search yields a bunch of stuff discussing ceramics and modeling cracks in ceramics. IPMech seems to be the Institute for problems in mechanics, part of the russian academy of sciences. I'm not sure ho legitimate that is though. Also I still don't know how they got into the account in the first place. The logs don't go far enough back to see the log in attempt that corresponds to the bask history.
  3. I have removed the crontab completely (there was nothing else in it) and deleted /var/tmp/.aw, and rebooted the server. I've checked the running processes for anything funny and everything looks legit. I've changed my password. Do I need to change my public keys as well?
  4. What else can/should I look for to help identify the intrusion mechanism?

Thanks

sirlark
  • 211
  • 2
  • 12
  • you should also consider changing all your passwords and check if there are ssh keys. if it is a shared box, you have to check if some users are "sudoers" while they shouldn't. – m0ntassar Mar 05 '12 at 11:38
  • @m0ntassar, I'd down vote your comment if it was possible, for the simple reason that the only sensible thing to do here is remove the system from the network, reinstall and restore data from a known good backup before the compromise. Fixing is never a good option when a system has been compromised, as you don't know what other hidden surprises might be lurking. – Bryan Mar 05 '12 at 12:42
  • possible duplicate of [My server's been hacked EMERGENCY](http://serverfault.com/questions/218005/my-servers-been-hacked-emergency) – Bryan Mar 05 '12 at 12:42
  • @Bryan: I'd argue against flagging as a duplicate. Partial duplicate, maybe. But my question is more specific: I've been hacked and had IPmech installed. What is IPmech, and what has it done? – sirlark Mar 05 '12 at 12:53
  • @sirlark, Had your question have specifically asked that, and nothing else, then yes, I'd agree, but you also asked for some generic responses of other things to look for, which have been covered in depth in the linked question. If you haven't already, your priority right now should be to follow the advise given in the linked question. – Bryan Mar 05 '12 at 13:07
  • @Bryan : that was a comment and not a complete answer. I do not agree when you say reinstall and restore data from a known good backup before the compromise since the server coudld have a vulnerability since it's first deployment that was not exploited until the compromise noticed : in this case there is no good known backup and the server admin could be restoring the vulnerability itself. There is no better option thant being aware of the vulnerability and fixing it. – m0ntassar Mar 12 '12 at 15:44
  • @m0ntassar, I suggest you read [this question and its answers](http://serverfault.com/questions/218005/my-servers-been-hacked-emergency) if you haven't already, as you can *never* be sure that you've removed all traces of the compromise. Nuke it, no exceptions. – Bryan Mar 13 '12 at 08:10

1 Answers1

3

I wrote an answer over on Security.SE a while ago with techniques for finding out how they got in. It's not a comprehensive answer as such a thing would fill an entire book.

The gist of it is: "Look in the logs; identify suspicious things (files, log entries) by timestamps".

In your case, use whatever logs you have to find his IP address (last -i or grep username /var/log/auth.log) and then look through all other logs (particularly web server logs if you run one) for that IP address. Searching all your logs for IPmech may also be useful. If you can find where he got it from you might be able to get a copy yourself and see what it does. My guess (based on my own Google search for IPmech) is that it was running an open proxy.

exotr looks like a typo of exit to me. He hit "o" instead of "i" and mashed "tr" instead of just "t". Especially since he successfully typed "exit" two commands later.

Cleaning up the server is all well and good but even if "everything looks legit", you can never be sure. Wiping and re-installing using a backup from before the compromise is a way you can be sure. See the copious advice in My server's been hacked EMERGENCY. It's good practice to do this anyway as backups aren't really backups until you have successfully restored from them.

Community
  • 1
Ladadadada
  • 25,847
  • 7
  • 57
  • 90
  • Can I ask where you found the info on IPmech? I tried googling "IPmech" and "IPmech hack" without much success. Thanks – sirlark Mar 05 '12 at 12:59
  • The second and third results for "IPMech" were on web-proxies.org and pointed to ipmech.info. That site has now been shut down. The free proxies "industry" is constantly looking for new IP addresses that have not yet been blocked by school administrators, hence the dubious methods of obtaining them. – Ladadadada Mar 05 '12 at 13:07
  • I think restoring from a back up is not very secure. The backup may have suspect files, in other words the backup may be tainted itself. You're better of rebuilding your system from scratch using the backup as a template. i.e.using /etc to change the configuration of the new system to how the old was, copy home directories (except the hacked one of course) and so forth, rebuilding should take a few hours at most. If your system is so customised rebuilding it from scratch is going to take days you need to look into ways of making it more standardised. – aseq Mar 05 '12 at 22:12
  • Restoring from backup could be the worst idea you can have on such situation unless you are sure you are not restoring the vulnerability it self that led someone to compromise your server. I'd go for a new fresh build/install with updated software – m0ntassar Mar 12 '12 at 15:50
  • 1
    Two people in a row missed the bit about restoring using a backup **from before the compromise**. Do I need to edit the answer to emphasise that a little more? As for the fresh install... do you really backup your kernel? Your system binaries? Of course you install those from bare metal. We're talking about restoring the unique stuff here that the OP wrote himself that *only exists in his backups*. – Ladadadada Mar 12 '12 at 23:00
  • @Ladadadada I'm not talking about system binaries and kernel, I'm talking about config files, file permissions,and buggy pieces of code containing vulnerabilities – m0ntassar Mar 13 '12 at 15:38