The problem
A client of mine asked me to take a look at his shared-hosting webserver for the following problem, but I'm stuck at finding out what's wrong. His server is being blacklisted by a lot of major blocking list such as CBL, Spamhaus and the blockling list from Outlook.com.
What I've tried already
I started by looking at the users in his DirectAdmin environment but I didn't find any users whom are sending more than couple of e-mails per day. I downloaded his exim log, took a look at the mail queue, but couldn't find anything out of the ordinary. Next thing I thought of was running findbot.pl from CBL, but it came up only with false-positives.
Another thing I tried was to change the sendmail_path in php.ini to log every e-mail that is being sent out via sendmail. However, everytime I changed the sendmail_path, all PHP processes started to hang. I tried different ways (MailCatcher, my own scripts), but every change made the processes hang. Really strange, but after I few tries, I moved on to the next step.
Next step: installing lsofand create an bash script that would print the output of lsof -i | grep smtp into a log file, every second, while printing the outpot of ps auxw to another log file every second. This gave me some valuable information, but I can't track the issue yet.
Where I'm stuck
So after letting it run for a couple of hours, I opened up both log files and saw a bulk of this rules:
lsof - logfile
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
exim 10921 mail 9u IPv4 2260427 0t0 TCP hostname-from-server.com:smtp->208.93.4.208:49711 (ESTABLISHED)
exim 10921 mail 10u IPv4 2260427 0t0 TCP hostname-from-server.com:smtp->208.93.4.208:49711 (ESTABLISHED)
When I look at the logfile and search for the PID that is mentioned in the lsof logfile, I see the following lines:
ps auxw - logfile
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
mail 1750 0.0 0.0 59032 1320 ? Ss Nov28 0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root 10909 0.0 0.0 103388 896 pts/2 S+ 17:44 0:00 grep mail
mail 1750 0.0 0.0 59032 1320 ? Ss Nov28 0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root 10917 0.0 0.0 103388 896 pts/2 S+ 17:44 0:00 grep mail
mail 1750 0.0 0.0 59032 1320 ? Ss Nov28 0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
mail 10921 0.0 0.0 61112 1792 ? S 17:44 0:00 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root 10923 0.0 0.0 103388 896 pts/2 S+ 17:44 0:00 grep mail
mail 1750 0.0 0.0 59032 1320 ? Ss Nov28 0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root 10931 0.0 0.0 103388 896 pts/2 S+ 17:44 0:00 grep mail
mail 1750 0.0 0.0 59032 1320 ? Ss Nov28 0:01 /usr/sbin/exim -bd -q15m -oP /var/run/exim.pid
root 10939 0.0 0.0 103388 896 pts/2 S+ 17:44 0:00 grep mail
The problem: there is nothing out of the ordinary with this line and I can't see which script, program of user called exim. When I take a look at the exim mainlog and rejectlog, I can't find the ip 208.93.4.208 nor can't I find any line at all around 17:44 (time according to the ps auxw log).
When I follow lines from the logfiles from e-mails that I send myself, I can find them in the mainlog from exim at exactly the time that is mentioned in the ps auxw log. It appears that, somehow, the spammails aren't logged in exim or are removed immediately after sending.
My questions
- I think I can solve my problem if I knew which script, program or user called the PID and invoked exim/mail. Does anyone have an idea?
- Is it possible that some other server, not ours, is sending out spam and is, for example, spoofing our IP-address? Maybe this is a very dumb question, but I'm curious, since it so easy to spoof headers.
Additional information
Via the provider-portal of Outlook.com, we managed to get one of the e-mail headers:
X-HmXmrOriginalRecipient: someone-who-received-our-spam@hotmail.com
X-Reporter-IP: [IP-from-some-who-flagged-as-spam]
X-Message-Guid: a2236172-9474-11e5-9c3a-00215ad6eec8
x-store-info:4r51+eLowCe79NzwdU2kR3P+ctWZsO+J
Authentication-Results: hotmail.com; spf=none (sender IP is [OUR-IP-ADDRESS]) smtp.mailfrom=minvituccia@blackberrysa.com; dkim=none header.d=blackberrysa.com; x-hmca=none header.id=minvituccia@blackberrysa.com
X-SID-PRA: minvituccia@blackberrysa.com
X-AUTH-Result: NONE
X-SID-Result: NONE
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTA7YT0wO0Q9MjtHRD0yO1NDTD02
X-Message-Info: 11chDOWqoTmjqhOzvWWho/vK8oL2x1FIoEm0Tn+r3D4Vy8IHo2wUnqS07yp2Fxclyw07ONZgeH1xFUrogbJOZz8Pfl5FrUXTGgolDal8+UhiPOrwCAKsLtRr0R42oH/Du2inmiSwuWc/pY9oiWRqLA5If7jw818pUulf3QP7m+wKn2HEVHAg2VBr+OqDk1w/hWWO68tIy1BSoE8QFSPMNXh31MYdKh4mif3jAqDU+0qWqWSAxPdE/A==
Received: from [our-hostname] ([our-ip-address) by COL004-MC2F4.hotmail.com with Microsoft SMTPSVC(7.5.7601.23143);
Thu, 26 Nov 2015 11:34:05 -0800
Return-path: <minvituccia@blackberrysa.com>
Received: (qmail 18660 invoked by uid 61081); 26 Nov 2015 20:52:03 -0000
Date: 26 Nov 2015 20:52:03 -0000
Message-ID: <20151126205203.18660.qmail@our-hostname.com>
From: "Meghann Gasparo" <minvituccia@blackberrysa.com>
To: "someone-who-received-spam-from-our-server" <someone-who-received-spam-from-our-server@hotmail.com>
Subject: You could strike all your limpid seed right into my love tunnel text me 1.970.572.00.14
Mime-Version: 1.0
Content-Type: text/html
Content-Transfer-Encoding: 8bit
Mime-Version: 1.0
Content-Type: text/html
Content-Transfer-Encoding: 8bit
X-OriginalArrivalTime: 26 Nov 2015 19:34:06.0061 (UTC) FILETIME=[69C119D0:01D12881]
<html><body>Throw some of your hot cum on my face, deep into my door<br>
or <a href="http://holidayextravaganza.org/wp-content/themes/">run my humps rubbed</a> once again.<br>
<a href="http://holidayextravaganza.org/wp-content/themes/">Watch my profile</a> to receive much more spicy fun or just sms right now 1-970-572-00-73</body></html>
--70969AA2-2F73-4465-8DF3-26DC57EA3967--
We don't use qmail as MTA. Needless to say, but the domain blackberrysa.com is not one of ours.
