9

I have some Linux servers, which from what I can tell are not vulnerable to the shellshock attack vector, but I am curious what the attack looks like in the logs. What does a successful attack look like in the appache2 logs? What would a successful attack look like in the system log?

What do failed attempts look like? I'd like to extend my black-list for shell-shock type attacks.

Mark
  • 34,390
  • 9
  • 85
  • 134
j0h
  • 301
  • 2
  • 6
  • 13
    A successful attack is not visible in the logs: good malware erases the log entries that would reveal it. – Gilles 'SO- stop being evil' Sep 26 '14 at 12:35
  • @Gilles at which point reading stdout would let you determine whether the files were deleted/updated. you can even recover the deleted logs if you're l33t enough – r3wt Sep 26 '14 at 12:37
  • 3
    @r3wt Reading stdout? No, you either made a typo or are misunderstanding something (but I don't know what). Unless you log to write-once storage, good malware can erase log entries (not erasing log files, that can complicate forensics but makes the presence of the malware obvious). – Gilles 'SO- stop being evil' Sep 26 '14 at 12:41
  • 1
    @Gilles , when you edit a file on unix, you are creating an entirely new file in a new location on the disk, by unlinking the old file. if it hasn't been overwritten on the disk yet, it is still there, waiting to be discovered. In the scenario you are talking about, you can unmount the filesystem and do a binary search on keywords you might find in a access log file. – r3wt Sep 26 '14 at 12:51
  • 2
    @r3wt I know, but why do you assume that the malware would use this method, instead of overwriting the existing file? (Which may or may not leave traces on the disk depending on the filesystem.) You can't assume that malware is going to do what's convenient to you. – Gilles 'SO- stop being evil' Sep 26 '14 at 13:20
  • Im assuming script kiddies are going to be hammering this, and that they wont be doing the best job deleting logs etc.. – j0h Sep 26 '14 at 13:43
  • 2
    A successful attack looks like what that attack does. If I connec to your server and just ls /usr/bin to see what's installed it won't look the same as if I scp /etc/shadow. You can also get logs for what you're already paying attention to, are you logging all accesses to your Web page? What about other entry points, are you monitoring them? This question doesn't make much sense without more details on your environment and what you have to protect. – Steve Dodier-Lazaro Sep 26 '14 at 13:45
  • @j0h do you believe that Skiddies don't have a button saying "cleanse logs"? – schroeder Sep 26 '14 at 15:36
  • @schroeder to cleanse logs you first need to know what logs are, and most skiddies don't know that. And given that every system is different it has to be done manually, I don't think a skiddie rootkit/PHP shell/whatever is good enough to adapt to each system's differences and find the path to each log file. –  Sep 26 '14 at 16:52
  • @AndréDaniel You grossly underestimate the scripts/shells that exist. I've seen them, used them, and tracked them on my honeypots. Fear the skiddies.... – schroeder Sep 26 '14 at 18:22
  • 1
    @schroeder Yea, though i walk through the valley of the shadow of I/O, i will fear no script kiddie. For though intelligence art with me; thy tls and thy fail2ban they comfort me. Though preparest a logfile before me in the presence of mine script children: though anointest my shell with code, my terminal runneth over. – r3wt Sep 27 '14 at 01:17
  • @Gilles you don't understand. Over writing the existing file calls `unlink` on the old one and stores the new version at an entirely new place on the disk, wherever it fits best. there is no such thing as "overwriting the existing file". no such thing exists. you might could do it in pure IO with C++, but to my knowledge there is no bash script that could do it, and that might be above the pay grade of most script kiddies. – r3wt Sep 27 '14 at 01:22
  • @r3wt You can modify parts of an existing file in place. It may be above the pay grade of script kiddies, possibly above yours too, but it's definitely not above mine. Security is not just about protecting against script kiddies. You can do it with shell tools (that's one of the few useful uses of `dd`), not that there's any point, an exploit can inject whatever payload it wants. As this long discussion is off-topic, please take further comments to chat, or ask a question on [so] or [unix.se] if you'd like to understand how files work. – Gilles 'SO- stop being evil' Sep 27 '14 at 01:31
  • @Gilles its definitely above my paygrade at this point, and thanks for the knowledge. Always the student, never the master. – r3wt Sep 27 '14 at 02:14
  • @Gilles If you only managed to compromise the apache user, it shouldn't be possible to erase the logs after the fact. Erasing the logs would require another vulnerability. But exploiting the vulnerability without leaving a trace in the logs in the first place would be possible. – kasperd Sep 29 '14 at 11:58

4 Answers4

14

This is an entry from my access_log of what my coworker did to my test machine... : 

10.11.12.13 - - [25/Sep/2014:16:00:00 -0400] "GET /cgi-bin/testing.cgi HTTP/1.0" 200 1 "-" "() { test;};echo \"Content-type: text/plain\"; echo; echo; /bin/rm -rf /var/www/"

In my error log I saw a lot of this: 

[Thu Sep 25 16:00:00 2014] [error] [client 10.11.12.13] /bin/rm: cannot remove `/var/www/icons/pie0.png': Permission denied

Bastard :-)

A well configured web-server won't be able to overwrite its logs, and in all but the tiniest environments, you should be using centralized logging to protect you from losing these kinds of log entries.

If you're successfully exploited, you may see lots of errors in the error_log which show failed attempts to access resources, execute programs or delete files. If they successfully escalate privileges, and you don't have off-site logging, then you may see no evidence.

mgjk
  • 7,535
  • 2
  • 20
  • 34
5

The most things you could notice at the moment are probes, scanning random servers for possible attack vectors, so its advised to patch.

A possible "attack" or at least a check could look a bit like this:

egrep "};|}\s*;" /var/www/logs/access*  xxx.xxx.xxx.xxx - -
[25/Sep/2014:12:12:12 +0100] "GET /cgi-sys/defaultwebpage.cgi
HTTP/1.0" 404 168 "-" "() { :;}; /bin/ping -c 1 xxx.xxx.xxx.xxx"

Where xxx.xxx.xxx.xxx is the IP address.

kalina
  • 3,354
  • 5
  • 20
  • 36
Lighty
  • 2,368
  • 1
  • 23
  • 36
5

If you have an Apache web server with a standard configuration, and a cgi scripts that makes use of bash, the log entry for a request may look identical in the following three cases:

  • A legitimate request is made.
  • A successful attack is performed.
  • An attempted attack fails because you have upgraded to a bash version that disables the vulnerable feature.

The only difference needed to turn a legitimate request into an attack is modification of one of the request headers, which is put into an environment variable.

If the User-agent header is used to perform the attack, it will be quite visible in the log access log, since that header is logged. But there are other headers which can be used to perform the attack, which are not logged by default.

If an attempted attack failed due to bash having been updated to the intermediate version that fixed the bug but still allowed functions to be specified through environment, you would see warnings in the error log. They may look like this:

[Thu Sep 25 20:46:51.483207 2014] [cgi:error] [pid 26424] [client 10.82.90.125:55631] AH01215: /bin/bash: warning: HTTP_ACCEPT_LANGUAGE: ignoring function definition attempt
[Thu Sep 25 20:46:51.483316 2014] [cgi:error] [pid 26424] [client 10.82.90.125:55631] AH01215: /bin/bash: error importing function definition for `HTTP_ACCEPT_LANGUAGE'
kasperd
  • 5,402
  • 1
  • 19
  • 38
-2

I saw the CISO of Mandiant speak recently, he said when he is asked if there was always a pattern of behaviour that distinguishes an attacker (malware) from valid activity, he replies that if the attacker behaved exactly the same as valid worker we should thank them for doing work for free for us. Clearly the attacker will behave differently in some way from a valid worker. This implies that if we monitor behaviour/activity on a system, we should be able to detect if any malicous activity has taken place. Unfortunately system logs are likley to be tampered with by an advanced attacker so as well as shipping out to syslog and all the usual methods, you must monitor the whole system behaviour in some way to look for anomolies rather than just scrape one log. Sorry if this is an inconvenient answer but APTs are rather inconvenient... I appreciate this post does not actually answer the original posted question and is therfore not 100% helpful, but I do want people to remember not to focus too closely on a specific log entry and do not forget to monitor/analyse general behaviour as well

TomH
  • 9
  • 1