How do I know how the hacker gained access to my machine?

Question

I own a dedicated machine and today I got an email from my host saying that my server will be blocked if I don't stop abusing the network. I am not a spammer or hacker. Someone was using my machine to do the mischief. They attached a note in the email showing the following.

##########################################################################    
#               Netscan detected from host  178.63.xxx.xxx               #
##########################################################################

time                protocol src_ip src_port          dest_ip dest_port

---------------------------------------------------------------------------
Sat Oct  4 11:24:09 2014 TCP  178.63.xxx.xxx 26217 =>      94.xxx.xxx.0 22
Sat Oct  4 11:24:09 2014 TCP  178.63.xxx.xxx 26217 =>      94.xxx.xxx.4 22
Sat Oct  4 11:24:09 2014 TCP  178.63.xxx.xxx 26217 =>     94.xxx.xxx.41 22
Sat Oct  4 11:24:09 2014 TCP  178.63.xxx.xxx 26217 =>     94.xxx.xxx.61 22
Sat Oct  4 11:24:09 2014 TCP  178.63.xxx.xxx 26217 =>     94.xxx.xxx.80 22

I checked my syslog and it mentions the following about the same time as above. Note the lines with **s. 'device entered promiscuous mode'

##################

Oct  4 11:20:01 v7 CRON[18962]: (cap) CMD (cd /var/tmp/ ; ./ps x >/dev/null 2>&1)
**Oct  4 11:24:34 v7 kernel: [7144175.421969] device eth0 entered promiscuous mode**
Oct  4 11:30:01 v7 CRON[19075]: (cap) CMD (cd /var/tmp/ ; ./ps x >/dev/null 2>&1)
**Oct  4 11:31:31 v7 kernel: [7144591.716996] device eth0 left promiscuous mode**
Oct  4 11:31:31 v7 kernel: [7144592.377159] ssh-scan[19234]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]

###################

I also noticed a lot of this:

#############

Oct  4 11:31:47 v7 kernel: [7144607.756445] ssh-scan[19178]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:50 v7 kernel: [7144611.372464] ssh-scan[19725]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:52 v7 kernel: [7144613.062509] ssh-scan[19326]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:53 v7 kernel: [7144614.091379] ssh-scan[19704]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:53 v7 kernel: [7144614.121336] ssh-scan[19747]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:53 v7 kernel: [7144614.377344] ssh-scan[19727]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:53 v7 kernel: [7144614.404263] ssh-scan[19712]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:54 v7 kernel: [7144614.557179] ssh-scan[19708]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:54 v7 kernel: [7144614.673588] ssh-scan[19709]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:54 v7 kernel: [7144614.678058] ssh-scan[19749]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]
Oct  4 11:31:54 v7 kernel: [7144615.195993] ssh-scan[19711]: segfault at 0 ip 0000000008048e33 sp 00000000ffa39110 error 4 in ssh-scan[8048000+c0000]

#########

Now what do I do? How do I know how the hacker gained access to my machine and how do I stop this from happening?

On another note, a similar incident happened a few months ago when my host said that my installation of elasticsearch could potentially be the problem of security hole. I uninstalled elasticsearch as I was not using it and changed the root password.

Any help in investigating this issue is much appreciated.

Thanks

UPDATE 1

I ran who -u and I see the following

root     hvc0         Jul 13 18:55  old          728
deployer pts/0        Oct  4 11:19 04:08       18836 (240.50.134.37.dynamic.jazztel.es) (IS THIS THE HACKER?)
root     pts/3        Oct  4 14:53   .         14794 (122.174.xxx.xxx) (THIS IS MY IP)

Answer 1

This is a difficult question to answer, because there's generally not a single thing that you should do. Rather you collect info from a variety of sources, see what you've got and follow those leads further. Still here's some ideas:

significant info I see in what you've given us includes:

• a cron job has been installed, and you know some times when it ran
• the cron job runs a program which is or was installed at /var/tmp/ps
• a program called ssh-scan has been running and has segfaulted repeatedly. It looks like that happened just after the cron job ran

There's not a whole lot to go on so far, but look for useful timestamps.

• creation and modification times of /var/tmp/ps
• modification time of the cron file that the job runs from.

With luck, those timestamps might correspond to the time when your machine was breached, in which case you know more about where to look in your logs for info about the actual breach.

Somewhere there is or was a program that the system identified as ssh-scan. Look for it with locate. It's possible fora program to rename itself after it runs, but in that case the program may have the text ssh-scan inside it, in which case you can maybe find it with grep.

Look through things like .bash_history files for mention of crontab, ps and ssh-scan. If the hacker left a record of command history, that might help you find it.

Look through the entries listed by last for anything unusual.

Look at the content of /var/tmp/ps. If it's a binary, look through the output of strings /var/tmp/ps.

• It's remarkably common for hackers to identify themselves in their programs, but it's not necessarily the case that the author of that file is the person who hacked your machine
• It may be that interesting looking stuff in that file, or the md5 checksum of that file can be located with google.

Hopefully something in the above will turn up some new leads.

The fact that the attacker has put a program in /var/tmp, and has been able to switch an interface's promiscuous mode on and off means they had root access. While you may be able to determine the point of entry, you are very unlikely to be able to ever be sure you've closed off their access. You should probably reinstall the server, and be very careful about what you are copying across to the new one. Understanding what the hacker did may be very useful in understanding what you should not copy across. Reinstall as much as possible from trusted sources outside the compromised system.

The hacker may still have access. They may try to erase evidence, possibly quite destructively, so make sure everything is backed up. Go through all cron and at entries for back doors. Using a firewall, lock down access as far as you can, including limiting access to as small a set of remote IPs as possible (eg only allow ssh access from designated IPs). If your host has privileged access to anything else, restrict it as far as possible.

Last but not least, consider getting professional help from someone with more experience.

[EDIT In response to your edit]

That may well be your attacker. It may also be an intermediary system already controlled by the hacker. Such arrangements can be elaborate and hard to track through, but there's also a lot of simple-minded hackers out there.

Once you have a backup, contact the administrators of jazztel.es. There's no whois for .es domains, but I think the address you want is abuse@jazztel.com. They can likely shut them down, and maybe help put things in front of the police.

[EDIT Another addition]

Amongst the various privileged access your server might have, If you are in the habit of forwarding your ssh agent connection to your server, then stop that now. It can easily be abused by someone with root access on your server, albeit only for as long as your ssh session is connected.