13

I recently got one Undelivered Mail Returned to Sender while sending my newsletter to one of my 1500 customers. My website uses a double-opt-in procedure to make sure, the user explicitly want to receive my newsletter.

The error message:

smtp; 554 ...
    Swisscom AG IP: 94.130.34.42, You are not allowed to send us mail. Please
    refer to xyz.com if you feel this is in error.

I got an example spam mail (from the mail provider of the receiving mailserver):

Received: from mail.com ([94.130.34.42])
        by smtp-27.iol.local with SMTP
        id itOWeYZ6O42IFitOWe35TR; Tue, 13 Feb 2018 03:54:09 +0100
From: "Servizi online - Poste Italiane" <posteitaliane@test123.it>
Subject: Abbiamo ricevuto una segnalazione di accredito
Date: Mon, 12 Feb 2018 11:32:03 -0500

The provider also stated, that my server seems to be hacked. He further stated, that "the recipient mail server has simply recorded the rDNS presented to it by the connecting IP, in this case mail.com ([94.130.34.42])" - which is definitely NOT as I configured my rDNS entry (mail.lotsearch.de) for my IP address. So if I understood rDNS correctly, the receving mail server queries the sender IP for a rDNS entry (94.130.34.42 => should resolve to => mail.lotsearch.de, which it definitely does, when I test it from my local machine via $ host 94.130.34.42).

How is it possible to spoof rDNS? I can not imagine any way how this can technically work (only with a man-in-the-middle attack somewhere in the infrastructure between the receiving mailserver and my server).

The provider mentioned also, that "it is likely that a machine connecting from my IP has been compromised and sending these messages via direct connections to the recipient mail sever (also known as direct MX)". What does direct MX mean? Someone stole or found leaked mail credentials to one of my mail accounts and used them for mail sending?

What I have done so far to make sure my server is NOT / will not be hacked:

  • searched the mail logs (var/log/mail*): nothing special in there
  • checked the ssh login logs (last, lastb): nothing unusual
  • checked if postfix is does relaying: no it does not (checked via telnet)
  • checked for malware via clamav: no results
  • installed and configured fail2ban for ssh, postfix and dovecot
  • installed the latest patches / updates for Ubuntu 16.04 (I do so every week)
  • checked if my IP address is on any blacklist: it is not
  • verified rDNS entry in the management console of my hosting provider: it is correctly set to mail.lotsearch.de.
  • changed passwords of all mail accounts
  • changed public keys for shell access

More important: There was no information about posteitaliane@test123.it in the logs. So if my server would have been misused by a spammer (f.e. because of leaked smtp credentials of one of the mail accounts) I would see that in the log files.

The last possibility I can think of is that an intruder placed malware at my server I didn't find yet.

How can I monitor outgoing mail traffic (per process and per port)?

Only monitoring outgoing port 25 would not help as this would only trap irregular mails sent via postfix, but not the mail traffic caused by a potential malware infection (if the malware uses another port than 25 for directly sending mails / communicating with recipient mail servers). If I monitor outgoing traffic on all ports I will get a way to huge log file which I cannot search through for suspicious activity efficiently.

EDIT - Added test for open relay:

$ telnet mail.lotsearch.de 25
$ HELO test@test.de
250 mail.lotsearch.de
$ MAIL FROM: test@test.com
250 2.1.0 Ok
$ RCPT TO:<realEmail@gmail.com>
454 4.7.1 <realEmail@gmail.com>: Relay access denied

EDIT - Running webapps

koseduhemak
  • 271
  • 3
  • 8
  • "If I monitor outgoing traffic on all ports"... Why? What other traffic is this mail server sending? Are you sure you have not configured an open relay? And no one with access to send mail on the server has leaked credentials? – Daniel Widrick Feb 14 '18 at 15:11
  • @DanielWidrick The mailserver is also used as a webserver, so traffic at 443 and 80 is also possible. I thought of some sort of malware placed on my server communicating directly with external mail servers (not via default mail server postfix). About open relay: I edited my question with the check I performed to assure that there is no open relay. Your last question is hard to answer as I don't know about the "health" of my customers computers (where mail accounts of my server are configured) or if they infected with malware / keylogger which obtained credentials etc. – koseduhemak Feb 14 '18 at 15:23
  • Disk is cheap. In your situation there's a valid case for keeping a year's worth of logs. Consider using syslog etc to ship them off the server directly. – Criggie Feb 14 '18 at 23:48

2 Answers2

13

Before I get to my suggestion, I want to comment a bit on something your provider said to you:

 Received: from mail.com ([94.130.34.42])

        by smtp-27.iol.local with SMTP
        id itOWeYZ6O42IFitOWe35TR; Tue, 13 Feb 2018 03:54:09 +0100

This does not indicate that the reverse DNS for 94.130.34.42 is (or was) mail.com. Rather, it indicates that the SMTP client sent mail.com in its HELO (or EHLO) line. (A well-configured mail server would have rejected this connection entirely, but that's on Swisscom, not you...) This line does not indicate any reverse DNS entry. If it did, it would have appeared within the parentheses. For example:

Received: from mail-io0-f197.google.com (mail-io0-f197.google.com [209.85.223.197])

In this case, the first hostname is what the mail server identified itself as in its EHLO. The second hostname is the reverse DNS recorded at the time the connection was made.

RFC 5321 section 4.4 explains the format of the Received: line, with a formal grammar.

In your case, no reverse DNS was recorded. Since your IP address has a PTR record, this may be because they didn't look it up, or there was a temporary DNS failure.

Now, it appears you run a web hosting service and have numerous web apps. If one of these is compromised, it may begin sending spam. These often make direct connections to remote mail servers by looking up their MX records and connecting to port 25, as if they were actually a mail server themselves, rather than delivering mail to the local mail spool or an authenticated mail service on ports 587 or 465 as legitimate web apps do.

One way I stop this is by implementing a firewall rule that prevents outgoing connections on port 25 unless the user is the mail server user. For example:

iptables -I OUTPUT -m owner ! --uid-owner postfix -m tcp -p tcp --dport 25 -j REJECT

Web apps can no longer deliver mail directly to remote SMTP servers, but must use the local mail spool or an authenticated mail service.

Community
  • 1
Michael Hampton
  • 237,123
  • 42
  • 477
  • 940
  • Thank you for your answer. How do I need to specify the `iptables` rule to let postfix and plesk user send emails (as I think the Plesk Panel does send mails directly and not via postfix). Is it also possible to configure crondaemon (my cronjobs) to send emails via smtp via postfix? I dont want to add the cron user to iptables (as another exception) as it would be more secure to let mail traffic where possible to go through postfix. Is it possible to let crontab use postfix for sending error logs? Should I put that in a new question here on serverfault? – koseduhemak Feb 15 '18 at 09:22
  • I don't know how to do this with Plesk. [We don't handle questions about Plesk here](https://meta.serverfault.com/q/8094/126632) anyway. – Michael Hampton Feb 15 '18 at 17:24
  • Ok, but if I want to specify multiple users which allowed to send data through port 25, can I just copy the iptables rule and add a second one with the other user or do I have to specify it within one rule? – koseduhemak Feb 15 '18 at 17:33
  • Probably not; you'd have to create a user chain, I think. – Michael Hampton Feb 15 '18 at 18:03
  • One thing about the provided iptables rule: Are you sure that we do not need to set the rule for user `root`? Because the master process of postfix is run by `root` in most cases. Or does the postfix master process spawn subprocesses using `postfix`-user to send emails / do stuff? I tried your iptables rule, emails could not be delivered... If I do `ps -ef | grep "postfix"` I see some subprocesses running by the `postfix`-user and one master process running by `root`... – koseduhemak Feb 21 '18 at 11:44
  • To answer my own question in the comment above: `iptables -I OUTPUT -m owner ! --uid-owner root -m tcp -p tcp --dport 25 -j REJECT` is the correct rule for `iptables` as my postfix (installed by PLESK is running as root (or at least the master process of postfix)). With this rule only root can sent emails through port 25 outgoing. – koseduhemak Mar 01 '18 at 14:38
7

In this day and age, trying to do your own mail server is, for the most part, a loosing battle and you are better off finding an affordable service. Having said that..

  • Look at your logs going to the provider that blocked you and see if you can find anything suspicious. It is possible, and happens often, that someone forgets they subscribed to your newsletter and marks you as spam. Then depending on the provider you can get on the provider's blacklist even though you have done nothing wrong.

  • Separate mass mailings from all your other email into two servers.

  • Keep logs for weeks at a minimum and better months. So anytime something happens you research.

  • Check your logs daily for similar situations from any provider and look into it daily, or faster.. The second you get blocked and if you keep trying to send it can get worse. You can go from a temporary block to a permanent block.. to getting reported to a blacklist.

  • Not sure how they implement it, but one thing I know many providers do for outbound mail services is that the second a provider/IP blocks an email then no other emails get tried to be sent. Ideally you want something like that. Because the second one gets blocked, sending more will just aggravate the issue.

Francisco1844
  • 260
  • 1
  • 8
  • you are definitely right, but services (as mailchimp f.e.) are very expensive... We send out every day a notification email (like ebay, where you can monitor search terms for your desired auction items). Given that we need to calculate for 1500 users with 30k mails per month at the moment and the user base is growing... But I will definitely try to go with your suggestions (especially preserving logs for a longer time) – koseduhemak Feb 14 '18 at 16:07
  • 4
    @mfuesslin Mailchimp would be the wrong platform to use. Mailchimp is an Email Marketing Service, what you need is a Transactional Email Service. Look into Mandrill (owned by the same people who own Mailchimp). It's $20 a month for a block of 25,000 emails. Not very expensive. Sending this many emails daily from your own IP address will only result in a high spam-box rate... it is a losing battle. You could hire an entire team to do nothing but tend to your deliverability rates all day every day, and still not be as good as using a Transactional Service. – SnakeDoc Feb 14 '18 at 19:10
  • 1
    People using serverfault.com should be capable of running a mail server; it's not that hard to do. That said, it doesn't seem that the mail server is at fault, it looks like some compromised web page that is directly sending the spam. – wurtel Feb 15 '18 at 10:49
  • 1
    @wurtel just because one has the knowledge of how to do something it doesn't mean it makes sense to do it. If you can find a service for X/month to do what you need and it takes you 4X/month worth of time/effort to do it yourself then it really does not make sense to do it yourself. – Francisco1844 Feb 15 '18 at 14:59
  • 1
    @wurtel Capable? Yes. Delivering consistently to inbox, sending 1500+ emails a day? Questionable, and probably a No. -- Nobody is saying you can't do it... only that to do it well, consistently, and over a long period of time, it'll cost you a lot more than $20 a month. – SnakeDoc Feb 15 '18 at 16:28
  • 2
    I've maintained such a server over 15 years, regularly sending out 30-50 thousand mailinglist messages in addition to hunderds of messages daily for multiple domains, and I rarely spend more than an hour a month (besides the regular aptitude upgrades). The server is serving multiple websites anyway, so there's no extra investment there. I'm a bit sad that people are advocating buying services to do things you can easily do yourself. Nothing wrong with a bit of learning along the way. – wurtel Feb 16 '18 at 11:54
  • 1
    @wurtel I'm not trying to argue or anything, but is spending an hour a month really cheaper than $20? Does your system provide you with Open and Click rates? Track and handle unsubscribes and both soft and hard bounces? Does it do all this per-email type or per-campaign? Do you know your inbox delivery rates, which email services have blacklisted your IP, which RBL's have picked up your IP and how to go about getting removed again? Do you know how your stats compare to others in your industry? -- If so, you're doing a lot more than 1 hour's work per month... or you're not getting good results. – SnakeDoc Feb 16 '18 at 17:49