1

Apache's error_log shows lines like the following:

--- snip ---
which: no ruby in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no locate in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no suidperl in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no get in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no fetch in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no links in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no lynx in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no lwp-mirror in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no lwp-download in (/sbin:/usr/sbin:/bin:/usr/bin)
which: no kav in (/sbin:/usr/sbin:/bin:/usr/bin)
--- end ---

The architecture is:

Internet -> Load Balancer -> Varnish -> Apache

There are several web servers behind the load balancer and I have checked at least one of them with rkhunter (link) and couldn't find anything suspicious.

Versions:

  • CentOS 5.7
  • Varnish 2.1.5
  • Apache 2.2.3
  • PHP 5.2.17

Does this mean that someone has executed the command which through Apache? How can that happen?

Thank you so much.

RichVel
  • 3,524
  • 1
  • 17
  • 23
Unai Rodriguez
  • 13
  • 2

1 Answers1

2

It is fairly clear you have been hacked. The commands mentioned would be of interest to a hacker who has found a vulnerability letting them execute commands as the apache user (probably through a web application vulnerability). For example, links and lynx would let the attacker download additional programs, as would lwp-*. These commands would never be executed by a legitimate web app so it must be an attacker.

See this thread for an attack with a similar signature - this is actually a privilege escalation.

The first thing is to take the system offline - a tradeoff decision but since you have no idea how completely the system is owned, it's the safe thing to do.

You should attempt to find out when you were hacked - e.g. restore backups from 4 weeks ago onto a separate system, and compare with a backup done last night. This should be a whole machine backup as the attacker might have got into root.

Once you know when you were hacked, you can restore from a backup that was made before the attack (but still keep the system offline so it can't be exploited again). The vulnerability is still there, so the attacker could get in again if the system was online - so you also need to quickly find and close the vulnerability - the complete machine backup from last night is important so you can find how they got in.

See How do I deal with a compromised server? for more complete advice on how to recover from a hack.

Good luck.

Community
  • 1
RichVel
  • 3,524
  • 1
  • 17
  • 23