3

Our server hosts over a thousand sites, and some of them seem to have been hijacked by malicious scripts. These scripts run actions normally performed by a legitimate user en masse, causing severe strees on our server and often requires us to restart to clear the load. We don't have a way to find out what they are. Recently these attacks have begun to affect our daily operations. Our error log file is 70mb in size with messages similar to the following:

[timstamp] [error] [client xx.xxx.xx.xxx] File does not exist: /path/favicon.ico (File exists. This is the majority of all log entries)
[timstamp] [error] [client xxx.xxx.xx.xxx] client denied by server configuration: /path/to/cron.php (This is my TOP concern)
[timstamp] [error] [client xxx.xx.xx.xx] Directory index forbidden by Options directive: /another/path
[timstamp] [error] [client xx.xx.xxx.xxx]  ALERT - canary mismatch on efree() - heap overflow detected (attacker 'xxx.xx.xxx.xxx', file '/path/to/index.php')
[timstamp] [error] [client xx.xx.xxx.xx]  Directory index forbidden by Options directive: /path/to/another/file_or_folder/
[timstamp] [error] [client xxx.xx.x.xx] Invalid URI in request GET /../../ HTTP/1.1
[timstamp] [error] [client xx.xxx.xx.xx] client denied by server configuration: /path/to/another/web/file/
Invalid URI in request GET mydomain.com HTTP/1.0

Our DB log file is above 5gbs in size.

My question is, what can we do to counter these threats? Is there a way to ban IPs based on certain behavior? We are still sifting through our logs and trying to determine a course of action. Any guides, references, or tutorials that could be provided would be greatly appreciated.

SFox
  • 133
  • 4

1 Answers1

4
  1. Upgrade your system. Ubuntu 9.04 hasn't gotten a security update in two years.
  2. client denied by server configuration: /path/to/cron.php - Don't worry about this one. The request was blocked by Apache's configuration and the attacker got a 403 Forbidden response.
  3. ALERT - canary mismatch on efree() - heap overflow detected (attacker 'xxx.xx.xxx.xxx', file '/path/to/index.php') - This can potentially be a major problem - a buffer overflow vulnerability can allow an attacker to take complete control of your system. On the other hand, it might just be that the attacker's attempts to take over your system simply triggered a bug in PHP.

The system may be compromised already; if in doubt, restore from backup. Then, upgrade this system to supported and current versions of the OS, which will also update your application packages. See if you're still having issues, and if so, work to counteract the buffer overflow by validating client input data thoroughly.

Shane Madden
  • 112,982
  • 12
  • 174
  • 248