0

Problem

I think my server is being used to send spam with sendmail, I'm getting a lot of mail being queued up that I don't recognize and my mail.log and syslog are getting huge.

I've shutdown sendmail, so none of it is getting out but I can't work out where it's coming from.

Investigation so far:

I've tried the solution in the blog post below and also shown in this thread.

It's meant to add a header from wherever the mail is being added and log all all mail to file, so I changed the following lines in my php.ini file:

mail.add_x_header = On
mail.log = /var/log/phpmail.log

But nothing is appearing in the phpmail.log.

I used the command here to investigate cron jobs for all users, but nothing is out of place. The only cron being run is the cron for the website.

And then I brought up all php files which had been modified in the last 30 days but none of them look suspicious.

What else can I do to find where this is coming from?

Mail.log reports

Turned sendmail back on for second. Here is a small sample of the reports:

Jun 10 14:40:30 ubuntu12 sm-mta[13684]: s5ADeQdp013684: from=<>, size=2431, class=0, nrcpts=1, msgid=<201406101220.s5ACK1cC011438@ubuntu12.pcsmarthosting.co.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jun 10 14:40:30 ubuntu12 sm-msp-queue[13674]: s5ACK1cC011438: to=www-data, delay=01:20:14, xdelay=00:00:00, mailer=relay, pri=571670, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s5ADeQdp013684 Message accepted for delivery)
Jun 10 14:40:30 ubuntu12 sm-mta[13719]: s5ADeQdp013684: to=<www-data@ubuntu12.pcsmarthosting.co.uk>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=32683, dsn=2.0.0, stat=Sent
Jun 10 14:40:30 ubuntu12 sm-mta[13684]: s5ADeQdr013684: from=<www-data@ubuntu12.pcsmarthosting.co.uk>, size=677, class=0, nrcpts=1, msgid=<201406101200.s5AC0gpi011125@ubuntu12.pcsmarthosting.co.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jun 10 14:40:31 ubuntu12 sm-msp-queue[13674]: s5AC0gpi011125: to=www-data, ctladdr=www-data (33/33), delay=01:39:49, xdelay=00:00:01, mailer=relay, pri=660349, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s5ADeQdr013684 Message accepted for delivery)
Jun 10 14:40:31 ubuntu12 sm-mta[13721]: s5ADeQdr013684: to=<www-data@ubuntu12.pcsmarthosting.co.uk>, ctladdr=<www-data@ubuntu12.pcsmarthosting.co.uk> (33/33), delay=00:00:01, xdelay=00:00:00, mailer=local, pri=30946, dsn=2.0.0, stat=Sent
Jun 10 14:40:31 ubuntu12 sm-mta[13684]: s5ADeQdt013684: from=<www-data@ubuntu12.pcsmarthosting.co.uk>, size=677, class=0, nrcpts=1, msgid=<201406101215.s5ACF2Nq011240@ubuntu12.pcsmarthosting.co.uk>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jun 10 14:40:31 ubuntu12 sm-msp-queue[13674]: s5ACF2Nq011240: to=www-data, ctladdr=www-data (33/33), delay=01:25:29, xdelay=00:00:00, mailer=relay, pri=660349, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s5ADeQdt013684 Message accepted for delivery)
Jun 10 14:40:31 ubuntu12 sm-mta[13723]: s5ADeQdt013684: to=<www-data@ubuntu12.pcsmarthosting.co.uk>, ctladdr=<www-data@ubuntu12.pcsmarthosting.co.uk> (33/33), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30946, dsn=2.0.0, stat=Sent
Ju

Further Investigation

Spotted 4 spam accounts registered in the past day, which is suspicious however all have normal user privileges.

There are no contact forms on the site, there are a number of forms and they take either filtered text input or plain text input.

Mail is still being queued up having switched the website to maintenance mode, which blocks out everyone but the admin.

Ok more investigation, it looks like the email is being send by my websites cron which runs every 5 minutes. However there are no cron jobs I've set-up which run more than once an hour and show on the website log so presumably someone has managed to edit my cron somehow.

Problem Over:

Turns out most of this was ignorance on my part. Cron tries to send an email when it runs. Because the cron was run by www-data, it tried to send it to www-data. The peculiar address was because I had never changed my dnshostname from the server default, which for some weird reason was pcsmarthosting.co.uk. (Weird because it's not at all related to my host.)

As I found out the format of the default address of www-data is hostname@dnshostname.

Copy of email:

V8
T1402410301
K1402411201
N2
P120349
I253/1/369045
MDeferred: Connection refused by [127.0.0.1]
Fbs
$_www-data@localhost
${daemon_flags}c u
Swww-data
Awww-data@ubuntu12.pcsmarthosting.co.uk
MDeferred: Connection refused by [127.0.0.1]
C:www-data
rRFC822; www-data@ubuntu12.pcsmarthosting.co.uk
RPFD:www-data
H?P?Return-Path: <�g>
H??Received: (from www-data@localhost)
        by ubuntu12.pcsmarthosting.co.uk (8.14.4/8.14.4/Submit) id s5AEP13T015507
        for www-data; Tue, 10 Jun 2014 15:25:01 +0100
H?D?Date: Tue, 10 Jun 2014 15:25:01 +0100
H?x?Full-Name: CronDaemon
H?M?Message-Id: <201406101425.s5AEP13T015507@ubuntu12.pcsmarthosting.co.uk>
H??From: root (Cron Daemon)
H??To: www-data
H??Subject: Cron <www-data@ubuntu12> /usr/bin/drush @main elysia-cron
H??Content-Type: text/plain; charset=ANSI_X3.4-1968
H??X-Cron-Env: <PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin>
H??X-Cron-Env: <COLUMNS=80>
H??X-Cron-Env: <SHELL=/bin/sh>
H??X-Cron-Env: <HOME=/var/www>
H??X-Cron-Env: <LOGNAME=www-data>
Community
  • 1
split_account
  • 169
  • 2
  • 11
  • I've done the suggestion in the highest voted answer there. It's the solution outlined in the [blog post](http://blog.rimuhosting.com/2012/09/20/finding-spam-sending-scripts-on-your-server/), but I'm not getting anything in the log I setup. Hold on I'll edit my answer to be clearer. – split_account Jun 10 '14 at 13:49
  • Not a php-head, but it looks like the post you are following will work iff the form is using php's mail() function - not all forms do that, some just spawn sendmail -t -bm, er even /bin/mail. – Alien Life Form Jun 10 '14 at 13:58
  • Anything in cron.{d,weekly,daily,hourly}? – Alien Life Form Jun 10 '14 at 17:02
  • 1
    I've narrowed it down (thanks to you guys) to my websites cron job, which is outputting the emails. I didn't understand that cron sends an email each time it completes. I'm still trying to discover why it has "www-data@ubuntu12.pcsmarthosting.co.uk" in it, which is what made me think spam in the first place, as opposed to, my cron is sending a completion email. – split_account Jun 10 '14 at 17:09
  • amazing how many people are happy to mark this as a duplicate and down vote it, while the referenced question addresses the issue assumed in the title of this question (which is wrong) it is quite clearly a different issue. That was apparent from the log entries in the original version of the question. It's quite possible (even likely) that this issue has arisen before, but the referenced question is not an example of that. – mc0e Jun 12 '14 at 02:42

3 Answers3

3

The spam, with 99% confidence, is coming from a poorly coded contact form deployed on a hosted web site. The 

ctl-addr=www-data

line is a dead giveaway (Hint: www-data is the name of the user the webserver runs as. It would be apache on a RH-like machine).

Shut down the webserver, look at its log, find out who's hitting the contact forms. It could be something more sophisticated (command/sql injection) or a complete website hijack/crack, but that's the first thing I'd check, absent other evidence.

Alien Life Form
  • 2,279
  • 2
  • 21
  • 31
  • Hmm so I put the website into maintenance mode, which blocks everything unless you're the admin user, but I'm still getting mail being queued up every 5 minutes or so. I have several forms, on the website (comment, create an advert), but none of them are contact forms. I also don't see anything aside from the usual requests to the pages with the forms on. – split_account Jun 10 '14 at 14:26
  • If the attacker has gotten hold of a vuln in your site, there is no guarantee it is not able to shortcircuit the 'maintenance mode' which is aopplication mandated. OTOH, looking at the web serever log you should now be able to see what's hitting it where (all else should be quiet). – Alien Life Form Jun 10 '14 at 14:35
  • It's completely quiet. Looking at the actual emails itself it looks like it's being sent by cron. – split_account Jun 10 '14 at 14:43
  • OK, what about: shut down sendmail; set up listeners on ports 25 & 587 (with, e.g,. nc -l); check the preocess connecting with netstat -p when you see the HELO line. – Alien Life Form Jun 10 '14 at 15:34
3

As noted by Alien Life Form, The mail looks to have originated from the www-data user. You haven't given enough info to conclude whether or not the problem is a "poorly coded contact form".

It may well be that some web application that is either being abused, or is misconfigured and is trying to tell you. The content of the emails should make that clear. Look at a few files in your mail spool if it's not otherwise easy to get a copy.

The very high frequency of these entries is notable, as is the fact that www-data seems to be mailing itself. Is this a mailing loop? That's particularly likely if you're not otherwise dealing with multiple php requests per second. Also www-data doesn't seem like the most likely target for spam.

Make sure that the www-data account is aliased to something sensible. eg you might use /etc/aliases (and run newaliases after editing that file). Depending on your distribution, that might be in a different location like /etc/mail/aliases.

If the problem is not a mail loop, then you presumably have web requests coming in at high speed associated with the emails being generated. You could capture some web traffic with tcpdump, and look through it (eg maybe with wireshark) to find the web requests involved (eg look for some content from the emails at the time). Unless it's coming via https, you'll then have the originating IP, and the URL.

You can likely also get the IP and URL from your http access logs, based on the timing of requests. How many URLs in there are accessed many times per second? Unless you have a very high traffic server, that's likely easier than capturing traffic. It also might allow you to quickly rule out web accesses being the source of the emails.

--

I see you've commented on Alien Life Form's answer that mail is being queued up 'every 5 minutes or so', which is a lot less frequent than in the log you provided. Given that, I'd be looking to line up the timing of those mail submissions with hits on the same URL in your access log (particularly look at POST requests, as they're more likely).

Are the requests exactly every 5 minutes? Do you have a 5 minute cron job?

Knowing a bit about the content of the emails would help quite a bit here in knowing which things to look for first.

mc0e
  • 5,786
  • 17
  • 31
  • Yep I'd just spotted that myself. It's being queued up by my websites cron. I have the website cron run every five minutes, (to catch my various cron jobs my website runs at different times past the hour). None of the cron jobs on my website are however run more than once per hour. So I guess they've somehow managed to hook something up to cron I can't see. – split_account Jun 10 '14 at 14:45
  • IF your cron script generates any output (on stdout or stderr) then that gets sent via email to the user that ran the cron script. From www-data to www-data seems about right for that. What happens to that mail next? Does it bounce back to sender? What does your cron job entry actually say? I'm thinking being hacked is not something you should be quick to assume here. You might want to set MAILTO in your cron file to direct mail where you want it to go (and then set it again after the cron entry in question or it will effect the rest of the entries that follow in the cron file). – mc0e Jun 10 '14 at 15:01
  • I set the MAILTO to an address that I own, however I don't ever get the email. You might be right that this isn't spam. The emails get stuck as queuing mail for delivery. In all honesty I'm now completely confused. Need to work out if my website cron job would send emails to itself and why. – split_account Jun 10 '14 at 15:27
  • 1
    The source of these mails likely isn't a mail issue, but to the extent that you do have a mail issue to solve, I'd think seriously about changing your mail software before doing much of anything else. Sendmail is a bit of a relic, and horrible to configure. It's the default on some systems, and sometimes that's fine, but as soon as you have to do anything to your mail configuration swap it out for something like postfix (my preference) or perhaps exim or qmail. I spent 10 years administering sendmail, and I really don't miss it. – mc0e Jun 10 '14 at 15:44
  • While mail is usually better, you could redirect your cron output to a file. eg: `23 * * * * www-data /usr/bin/drush @main elysia-cron >>/path/cron_output 2>&1 – mc0e Jun 10 '14 at 15:44
  • Ah I think my problem is didn't understand that cron tries to send an email output everytime it runs. Sendmail only got installed when I ran tiger to audit security so presumably it was just failing silently before then. It was the "www-data@ubuntu12.pcsmarthosting.co.uk" that made me think it was spam. Do you have any idea why that would be in the mail? – split_account Jun 10 '14 at 16:13
  • www-data is the user that's running the cron job, and ubuntu12.pcsmarthosting.co.uk is the default domain added by your MTA software (ie sendmail), which might be taken from `hostname -f` or it might be configured otherwise in your mail configuration. – mc0e Jun 12 '14 at 02:35
  • Yep spot on. the format seems to be hostname@dnshostname by default. I'd not changed my hostname because I'm not a great sysadm and because the default name wasn't related to my hosting provider I panicked and assumed the worst. Thanks very much for the help, you saved me a huge amount of pain. – split_account Jun 13 '14 at 11:57
0

Commands and useful places I looked to debug this problem (aside from everything in the answers above which is a given)

Add headers to php to track all email shown in this blog post and this server fault thread.

Check all the cron jobs being run with this command shown here.

You can find all of sendmails emails in /var/spool/mqueue or /var/spool/mqueue-client

You can view your mail queue with mail -q.

The from address for sendmail emails if it's being sent from www-data will probably be your machine hostname@dnshostname with the commands:

hostname
dnshostname

You can shutdown all email from a cronjob by adding >/dev/null 2>&1 to the end of it: e.g

*/5 * * * * /usr/bin/drush @main elysia-cron >/dev/null 2>&1

to test where it's coming from.

You can set the variable

MAILTO="me@me.com"

At the top of a cron job to have all email from that particular cron sent to you.

Community
  • 1
split_account
  • 169
  • 2
  • 11