3

The server of a customer has been hacked last night. It is a standard debian distribution with mysql, apache and ssh for remote access.

The way it has been hacked is very peculiar. The standard URL showed a deface page with link to nethack.alt.org. I have searched entire drive and could not find the files that would belong to this defaced page (bg.jpg ie.). Also, the ssh service and mysql were not accessible. Simple reboot did not help.

I see two possible scenarios:

  1. Hackers somehow installed a parallel system that got booted instead of the original debian.
  2. They somehow redirected all traffic to some other server (ip adress too, not just dns)

Does someone have experience with this kind of attack? Where can I look for compromised files?

Note: I will reinstall the entire system from scratch, but I would like to know how they came in, so I can prevent it in the future.

Edit: Additional info. This line is the last in my syslog when the rogue system was booting instead of the real one.

Jan 21 08:33:09 Debian-76-wheezy-64-minimal kernel: [   17.223513] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

On a normal start the mysql is started at that point:

Jan 21 09:12:12 Debian-76-wheezy-64-minimal mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Bojan Hrnkas
  • 133
  • 3
  • Could they have changed the apache config to redirect to a new page? The post lacks seriously in details, what version of Debian, what were the pages running, web server used (Apache, nginx), any CMS (joomla, wordpress), versions of Joomla, regular updates made, any firewall, local users? – Rui F Ribeiro Jan 21 '16 at 10:40
  • @RuiFRibeiro there were no changes to the original system as I could tell. After I rebooted the system using a special console, everything was back to normal. See additional info (nf_conntrack). I believe that is the culprit. – Bojan Hrnkas Jan 21 '16 at 10:42
  • please display `uname -v` and `grep ^Start-Date /var/log/apt/history.log | tail -1` – Rui F Ribeiro Jan 21 '16 at 10:45
  • The nf_conntrack is part of iptables and usual. – Rui F Ribeiro Jan 21 '16 at 10:50
  • uname: #1 SMP Debian 3.2.60-1+deb7u3 history.log is empty I know that nf_conntrack the part of kernel is, but it is suspicious that it gets started and then the syslog ends at that point. In a normal boot, the mysql should be started instead at that moment. Can nf_conntrack be exploited to redirect all trafic to some other server? – Bojan Hrnkas Jan 21 '16 at 10:55
  • History.log empty is a bad sign and security updates are not up to date. – Rui F Ribeiro Jan 21 '16 at 10:59
  • the machine seems to be not updated for ages, vulnerable to IPv6 bugs, ICMP bugs at least and just by running that kernel... ssh bugs and more...they could have got in by anywhere. I am not surprised there are differences in behaviour, they should have modified things around. kill it. – Rui F Ribeiro Jan 21 '16 at 11:01
  • So, basically, they cleaned up after themselves? Ok, thank you. – Bojan Hrnkas Jan 21 '16 at 11:07
  • 2 years without updates? Yep, there are clues they cleaned up things, certain logs should not be empty. They installed stuff for sure. – Rui F Ribeiro Jan 21 '16 at 11:10

1 Answers1

3

From the post and ensuing dialogue:

The machine is a Debian wheezy / 7.0 that aparently has not seen updates for around two years. There are logs lacking, that indicate the attackers cleaned up traces of their activity. The OP complains of a divergent behaviour, which corroborate the last hypothesis. [ and the compromised page ]

As for the length of time without updates, there have been multiple holes, remote and local, at kernel level, and in daemons. Vulnerabilities in IPv6, ICMP, ssh and so on. The attackers could actually have gotten by pretty much anywhere in the system.

It is strongly recommend to kill this machine.

Hackers also often install kernel modules to hide their tracks. If you wish to perform a post-morten analysis, boot it with a Live CD/DVD and dump the disk with dd to a DVD for analysing it.

As for recommendations for the future:

  • Rules to use new versions of the operating system;

  • a sensible update policy. No machine, as the best as it can be configured, will withstand an attack without having regular security updates;

  • uphold the Unix principle of installing the minimum services needed;
  • Creating firewall policies, just expose to the Internet the essential ports;
  • Install a WAF application, for instance, modsecurity if using Apache;
  • update your CMSs regularly, Joomla, wordpress and Drupal have more holes than a swiss cheese;
  • Consider looking into containers, namely Docker;
  • Follow standard programming practices to avoid layer-7 attacks like SQL injections;
  • Monitor for usage and unusual activity;
  • Keep regular backups.
Rui F Ribeiro
  • 1,736
  • 8
  • 15