False IP blacklisting on CBL and Spamhaus ZEN

Question

I've gone thru this: Spamhaus XBL keeps adding my IP

I'm managing the server 23.239.30.81 on Ubuntu using Postfix.

Since last 6 months or so I started forwarding all my:

• Cron Daemon logs
• Postfix SMTP errors
• Drupal website error logs
• Copy of subscription emails etc

to my one hotmail email.

But two weeks ago it was put on Spamhaus ZEN & CBL blacklist. But now I've changed the email from hotmail to one privately managed MS Exchange mail server.

But still after 3 days it is added back to blacklist.

Since a year I also have email monitoring setup using a cron script using pflogsumm which informs me if emails per day sent exceeds 300. But there aren't that many and just now I've checked in evening 6.30 pm and I can see just 93

Now I've added "always_bcc=myemai@privatecompany.com" in Postfix main.cf so that I can see all outgoing emails from this server. Here are the snapshots of those emails.

There isn't any spam .. I can see those( in addition to above) are:

• Contact us messages
• forum replies
• Account activation emails
• Mysql database backups
• Advertising reports etc

I'm forwarding the emails to a private email servers, how come Spamhaus ZEN & CBL can see those emails(except for forum subscriptions & replies) to blacklist this IP again?

I've emailed to cbl[@]abuseat.org but I've got only seemingly automated reply to check the system for viruses etc.

I've also run the system using clamAV antivirus.

All the forum replies and subscription and activation emails contain unsubscribe information. However the error logs do not.

What could I be missing?

Update: I've restricted emails to be sent only through Postfix: Firewall rule to only allow Postfix to send email through SMTP on port 25 and I do have copy of all the emails and none is spam. However it has been relisted the 4th time after some 15 hours.

Today(on 8th Dec) I've this reply from CBL:

The IP 23.239.30.81 is infected with spamware, most recently detected at:

2015:12:04 ~14:30 UTC+/- 15 minutes (approximately 3 days, 3 hours, 59 minutes ago)

This host HELOed as [127.0.0.1] ... Please correct that.

Answer 1

I would highly recommend to go through the CBL and Spamhuas websites again, because they have all the information you need to start with troubleshooting and safeguarding your server. The information there can help you understand how the blacklisting process works and why one gets listed and also advices on keeping server safe to avoid blaklisting.

I'm just going to quote a few important parts from CBL, the rest you can check for yourself. The point is, since you are getting listed again and again, your server is most probably compromised and it is not related to your postfix. Now you need to investigate and find out the possible cause. It can be a rootkit or a trojan or spambot or just another malicious script. You need to do a complete scan of your system for possible issues. Once you find the real cause, then you can resolve the problem and can take necessary steps to avoid it happening again.

Here is from CBL:

What is the CBL?

The CBL takes its source data from very large mail server (SMTP) installations. Some of these are pure spamtrap servers, and some are not.

The CBL only lists IPs exhibiting characteristics which are specific to open proxies of various sorts (HTTP, socks, AnalogX, wingate, Bagle call-back proxies etc) and dedicated Spam BOTs (such as Cutwail, Rustock, Lethic, Kelihos etc) which have been abused to send spam, worms/viruses that do their own direct mail transmission, or some types of trojan-horse or "stealth" spamware, dictionary mail harvesters etc.

I'm running Linux (FreeBSD, OpenBSD, UNIX...) and CANNOT be infected with a virus!

While it is perfectly true that UNIX-like operating systems are almost NEVER infectable with Windows viruses, there are a number of virus-like things that UNIX-like systems are susceptible to. For example:

• Windows emulation software (eg: VMWARE or Wine) are just as susceptable to infection as native Windows. In fact, it's probably somewhat more likely that an emulator instance of Windows gets infected, because the fact that it's running under another O/S can lead to a false sense of security, and emulator instances are less likely to be protected with a full anti-virus suite.
• Open proxies (eg: insecure Squid configurations) leading to open proxy spamming.
• Web server vulnerabilities or compromises. For example, the DarkMailer/DirectMailer trojan is injected via FTP (using compromised user's userid/passwords) onto web servers, and thereupon is used to send very larger volumes of spam. Virtually all web servers are susceptible to this if they permit upload of content from the Internet.
• Application vulnerabilities: many applications have security vulnerabilities, particularly those associated with PHP on web servers. Eg: older versions of Wordpress, PHPNuke, Mamba etc. Some of these vulnerabilities are to the extent that a malefactor can install a full proxy/trojan spamming engine on your machine and control it remotely. Through this, they can set up spamming engines, open proxies, malware download and spam redirectors. Watch out for strange directories being created, particularly those starting with a "." in /tmp. Check for this by doing an "ls -la" in /tmp, and look for directory names starting with "." (other than "." and ".." themselves).

For troubleshooting and protection

• It is CRITICALLY IMPORTANT that all web-facing applications or application infrastructures (Wordpress, Joomla, Cpanel, etc. etc.) are kept fully patched and up-to-date. Furthmore, userid/passwords and other credentials for logging into such systems should be highly protected, require strong passwords and changed as frequently as practical/feasible.

• Such sites should consider continous monitoring of web, ftp and other subsystems.

• Rootkits are where a malicious entity has installed software on your machine and buried it in such a way that the normal system utilities cannot find it. In some cases they replace the normal system utilities with hacked versions that won't show their tracks.

• Check that you have good remote login-capable passwords (eg: telnet, FTP, SSH), inspect your logs for large quantities of failed/SSH/telnet login attempts.

• Consider running a "system modification" detector such as Tripwire or rkhunter. Tripwire is designed to detect and report modification to important system programs. Rkhunter does what Tripwire does, but looks for specific rootkits, insecure versions of system software and more. Not all viruses are windows binaries. Some viruses/worms are in application-level files using non-binary programming techniques (such as macro viruses, Java, PHP or Perl). These can be truly infectious cross-platform.

More on MailServer in CBL: Mail Server in CBL

From Spamhuas:

What is "proxy hijacking"? What do I need to know about proxies?

What is a "honeypot" or "proxypot"? What is a "proxy hijack source" or "C&C"?

Answer 2

I'm coming in late, but:

1. You are no longer on the Spamhaus list.
2. CBL has removed you, but explains:

IP Address 23.239.30.81 is not listed in the CBL.

It was previously listed, but was removed at 2015-12-07 18:46 GMT (1 days, 5 hours, 1 minutes ago) At the time of removal, this was the explanation for this listing:

This IP is infected (or NATting for a computer that is infected) with the kelihos spambot. In other words, it's participating in a botnet.

If you simply remove the listing without ensuring that the infection is removed (or the NAT secured), it will probably relist again.

When this happened to us, it was because a client brought an infected laptop to our network. The guest network is segregated from our main network, but it's still NATed through the same IP as our regular traffic, including our mail server.

It looks like you're using shared hosting. If you're sharing an IP it's possible that one of the other hosts is actually the one that's infected. That might be a case for hosting support, if so. But the good news is that you're no longer on either list.

Answer 3

If you are on CBL, this is likely not a problem with your Postfix. CBL does not list open relay MTAs but rather servers which are "spam proxies". The latter often come from web scripts or other programs that were uploaded exploiting some security holes.

As this is a LAMP stack, I'd start from checking if there is no PHP backdoor script. Check if your data directories which are writable by PHP (or other scripting engines if you have any) contains any script files. Normally, only selected directories of your PHP web applications should be writable other that by the root user, make sure it's configured that way. On the web server level make data directories to only serve files with safe extensions (images, documents, etc., never any scripts). Refresh (or, if possible, upgrade) your web application script files.

It won't hurt to also secure port 465/TCP like you did with 25/TCP.

Answer 4

@Katherine-vilyard has given you very specific information (from CBL) about why your server is identified as sending spam: It is identified as sending spam due to infection with the kelihos spambot.

You say you have firewalled outgoing port 25 so only postfix can use it. We can't check if that's correct unless for example you give us the output of sudo iptables -L -v. Maybe you have made a mistake there, or maybe the mail is in fact passing through postfix. Maybe there's a spam bot on your system that has sufficient permissions on your server to circumvent this. e.g. by running as postfix. Maybe you're relaying spam rather than acting as the source. It sounds like you're aware of the problems of forwarding spam, and on the lookout for that.

IT sounds like you might be in a position to restrict the destination host for connections over port 25? Do so if possible.

If the mail is passing through postfix, using your usual configuration, it will be logged. You have a very specific timestamp to look at (from CBL). You should set about finding that email, or at least get all the info you can about it from your mail logs. The bit about HELO 127.0.0.1 should be an important clue - does your postfix server usually do that? If not, it's likely the mail didn't go via your postfix server. I'm guessing that's the case.

You should not assume that your postfix server recording its outgoing mail is enough. How about capturing all traffic on port 25 with tcpdump -i any -w dumpfile port 25 and then going through that? Looking through that for a timestamp that CBL identifies would be useful. Look for mail connections to unexpected destination. Compare the times of connections to the the timestamps you see in your logs. (Yes, I know I'm glossing over the detail of how to process the dumpfile. Wireshark might be useful, and also ngrep).

As I understand it, Kelihos infects windows PCs. That implies that your server is acting as some sort of relay rahter than a source. Are you running a VPN through it? Are you using it to relay your own outgoing mail? Are you sure it's not usable as a relay by any other host?