2

My server is sending lots of spam and I searched for the problem for hours now. After googling I found a forum where they talked about this and mentioned to dig into the exim log, so I did and found that emails were sent from: [username]@vps1.[hostname].[tld]. In the forum they said the e-mails were probably sent from my server because this is not a used e-mail address. They also mentioned to dig into php logs.

I've tried this but couldn't find anything, so via e-mail headers I'm now trying to detect the script sending all these e-mails. Which is were I'm stuck now.

I've changed php.ini by adding the following rules:

mail.add_x_header = On
mail.log = /var/log/phpmail.log

Also I've added exim.conf by adding this line:

+arguments \

Restarted exim and apache, but I don't see any X-PHP-Script headers in the exim log and the php mail log isn't created.

Only thing I see is an X header in the exim log:

X=TLSv1:RC4-SHA:128

Can anyone tell me what to do next?

EDIT

Here are some lines from the exim log:

bash-3.2# cat /var/log/exim/mainlog | grep 1W9FsC-0003qq-S2
2014-01-31 16:19:16 1W9FsC-0003qq-S2 <= instijl@vps1.xxx.nl U=instijl P=local S=816 T="Re:  It's good to see you," from <instijl@vps1.xxx.nl> for richisone@bigpond.com
2014-01-31 16:19:16 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9FsC-0003qq-S2
2014-01-31 16:19:17 1W9FsC-0003qq-S2 ** richisone@bigpond.com F=<instijl@vps1.xxx.nl> R=lookuphost T=remote_smtp: SMTP error from remote mail server after initial connection: host extmail.bigpond.com [61.9.168.122]: 554 nskntcmgw02p BigPond Inbound IB103. Connection refused. 141.138.199.65 has a poor reputation on the Cloudmark Sender Intelligence (CSI) list. Please visit http://csi.cloudmark.com/reset-request/?ip=141.138.199.65 to request a delisting.
2014-01-31 16:19:17 cwd=/var/spool/exim 7 args: /usr/sbin/exim -t -oem -oi -f <> -E1W9FsC-0003qq-S2
2014-01-31 16:19:17 1W9FsD-0003r9-H9 <= <> R=1W9FsC-0003qq-S2 U=mail P=local S=2006 T="Mail delivery failed: returning message to sender" from <> for instijl@vps1.xxx.nl
2014-01-31 16:19:17 1W9FsC-0003qq-S2 Completed

bash-3.2# cat /var/log/exim/mainlog | grep 1W9FsC-0003qc-M7
2014-01-31 16:19:16 1W9FsC-0003qc-M7 <= instijl@vps1.xxx.nl U=instijl P=local S=822 T="Re:  It's good to see you," from <instijl@vps1.xxx.nl> for richisingh7710@gmail.com
2014-01-31 16:19:16 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9FsC-0003qc-M7
2014-01-31 16:19:17 1W9FsC-0003qc-M7 ** richisingh7710@gmail.com F=<instijl@vps1.xxx.nl> R=lookuphost T=remote_smtp: SMTP error from remote mail server after end of data: host gmail-smtp-in.l.google.com [173.194.65.26]: 550-5.7.1 [141.138.199.65      12] Our system has detected that this message is\n550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to Gmail,\n550-5.7.1 this message has been blocked. Please visit\n550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131 for\n550 5.7.1 more information. y48si18631040eew.58 - gsmtp
2014-01-31 16:19:17 cwd=/var/spool/exim 7 args: /usr/sbin/exim -t -oem -oi -f <> -E1W9FsC-0003qc-M7
2014-01-31 16:19:17 1W9FsD-0003r1-BS <= <> R=1W9FsC-0003qc-M7 U=mail P=local S=2146 T="Mail delivery failed: returning message to sender" from <> for instijl@vps1.xxx.nl
2014-01-31 16:19:17 1W9FsC-0003qc-M7 Completed

bash-3.2# cat /var/log/exim/mainlog | grep 1W9Frw-0003oS-Gd
2014-01-31 16:19:00 1W9Frw-0003oS-Gd <= instijl@vps1.xxx.nl U=instijl P=local S=823 T="FW:  Yo" from <instijl@vps1.xxx.nl> for ketabatgooll@yahoo.com
2014-01-31 16:19:00 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9Frw-0003oS-Gd
2014-01-31 16:19:02 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [98.136.217.203]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:03 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [98.136.216.26]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:04 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [66.196.118.36]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:06 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [98.138.112.33]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:07 1W9Frw-0003oS-Gd SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [66.196.118.35]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html
2014-01-31 16:19:07 1W9Frw-0003oS-Gd == ketabatgooll@yahoo.com R=lookuphost T=remote_smtp defer (-45): SMTP error from remote mail server after MAIL FROM:<instijl@vps1.xxx.nl> SIZE=1866: host mta6.am0.yahoodns.net [66.196.118.35]: 421 4.7.1 [TS03] All messages from 141.138.199.65 will be permanently deferred; Retrying will NOT succeed. See http://postmaster.yahoo.com/421-ts03.html

bash-3.2# cat /var/log/exim/mainlog | grep 1W9Frg-0003mP-S6
2014-01-31 16:18:44 1W9Frg-0003mP-S6 <= instijl@vps1.xxx.nl U=instijl P=local S=814 T="call me" from <instijl@vps1.xxx.nl> for ket@web.de
2014-01-31 16:18:44 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1W9Frg-0003mP-S6
2014-01-31 16:18:45 1W9Frg-0003mP-S6 => ket@web.de F=<instijl@vps1.xxx.nl> R=lookuphost T=remote_smtp S=837 H=mx-ha03.web.de [213.165.67.104] X=TLSv1:AES256-SHA:256 C="250 Requested mail action okay, completed: id=0Le6s0-1VUM4v1jno-00pvEX"
2014-01-31 16:18:45 1W9Frg-0003mP-S6 Completed
Tim Baas
  • 281
  • 2
  • 5
  • 15
  • There don't seem to be any log entries showing these mails actually coming in to the mail system. Did you miss them, or did exim just not log them? – Michael Hampton Jan 31 '14 at 15:10
  • @MichaelHampton I don't really know which parts to copy and show here, there are a lot of mails bouncing back because the recipient doesn't exist. I've added some other lines, should I `grep` the lines of a certain ID? – Tim Baas Jan 31 '14 at 15:17
  • Try and get everything relating to three or so of the email addresses, above all try and get the first "Recieved From" lines.. or even better, pick a transaction ID, like `1W9Fkn-0002lF-QY` – NickW Jan 31 '14 at 15:27
  • @NickW I've added the logs of 4 spam e-mails. Is this helpful? – Tim Baas Jan 31 '14 at 15:33
  • The "U=instijl" tells you that whatever is sending the emails is running as user instijl. First see if the user is logged in with a shell. Second use 'ps aux' to find if any processes are running by that user. Third, look in your apache access logs to see what traffic is being sent to apache at the exact same time as 4 mails above. I suspect you have an insecure "send me feedback" form that is being abused (insecure because you allow incoming http request to set the sender, the recipient, and the message body). Look to apache pages for this problem. – Todd Lyons Jan 31 '14 at 15:40
  • @ToddLyons Well, that is enough information to get me going. – Tim Baas Jan 31 '14 at 16:02
  • @ToddLyons So, going down the road I found that no user except for me was shell logged in. Two dovecot/imap and one dovecot/pop3 processes. And the problem now is that my Access Log is empty.. In `httpd.conf` it has the following line regarding the access log: `CustomLog /var/log/httpd/access_log common`, isn't this what it should be? – Tim Baas Jan 31 '14 at 16:17
  • If the virtual host that is serving and accepting this request doesn't have its own access log entry, it won't log to the general access log (which is likely what you found). Find the specific section which is answering requests for that user and add the access log entry (or if it's already logging, figure out the filename). – Todd Lyons Jan 31 '14 at 17:11
  • 1
    Found the access logs for specific (virtual)domains, checked a few logs and found a log displaying 90% of POST's to a php file. Renamed this file and no spam is sent anymore.. I'm putting all this in an answer now.. Thanks for the help! – Tim Baas Jan 31 '14 at 17:13
  • Another thing you can do is 'yum install ngrep' (may be in external repo such as epel) and run 'ngrep -n -q port 80' and see what traffic is coming in. A more specific command which only shows incoming requests would be "ngrep -q -s 240 'GET|POST' port 80". Adjust the 240 up or down if you want to see more or less of the request, or omit it if you want to see the full request. – Todd Lyons Jan 31 '14 at 17:14
  • Glad you fixed it! As an aside, if you run 'httpd -S', apache prints out basic virtual host configuration to help you more easily find where in the config file that section is controlled/configured. – Todd Lyons Jan 31 '14 at 17:16

1 Answers1

2

Summary of TroubleShooting steps

The "U=instijl" shown from your /var/log/exim/mainlog excerpt tells you that whatever is sending the emails is running as user instijl. First see if the user is logged in with a shell. Second use 'ps aux' to find if any processes are running by that user. Third, look in your apache access logs to see what traffic is being sent to apache at the exact same time as 4 mails above. I suspect you have an insecure "send me feedback" form that is being abused (insecure because you allow incoming http request to set the sender, the recipient, and the message body).

If the virtual host that is serving and accepting this request doesn't have its own access log entry, it won't log to the general access log (which is likely what you found). Find the specific section which is answering requests for that user and add the access log entry (or if it's already logging, figure out the filename). If you run 'httpd -S', apache prints out basic virtual host configuration to help you more easily find where in the config file that section is controlled/configured.

Another thing you can do is 'yum install ngrep' (may be in external repo such as epel) and run 'ngrep -n -q port 80' and see what traffic is coming in. A more specific command which only shows incoming requests would be "ngrep -q -s 240 'GET|POST' port 80". Adjust the 240 up or down if you want to see more or less of the request, or omit it if you want to see the full request.

Todd Lyons
  • 2,006
  • 16
  • 12
  • I wanted to do exactly this, great work! You may want to add where "U=instijl" is (/var/log/exim/mainlog).. Thanks! – Tim Baas Jan 31 '14 at 17:59