2

In reference to the recently publicized Exim vulnerability CVE-2019-10149, I am running supposedly patched Exim v. 4.90_1 (built June 4th, 2019) on Ubuntu 18.04.2 LTS.

Although it is supposedly patched, according to Canonical, I'm getting a lot of exploit attempt that are ending up as Frozen messages.

This is somewhat worrysome.

Is there a way to reject messages for recipients starting with a '$'?

And how could I test to be sure these exploits aren't working?

jdmayfield
  • 271
  • 2
  • 11
  • These attempts usually come from the same sender (easy check : `for ii in $(sudo exiqgrep -z -i ); do sudo exim4 -Mvh $ii; done|grep host_add`). Wouldn't it be sufficient to blacklist the corresponding IP ? You'd have better control about what is going on on your server. – Skippy le Grand Gourou Jun 16 '19 at 22:31
  • On one server, mostly a certain Ip, but others have more. It is possible there are variants of the exploit that could be more effective, especially if combined with other exploits or techniques. Given the possible consequences, I'd rather be safe. – jdmayfield Jun 17 '19 at 03:51
  • Update, I have now been seeing attempts from a large number of IP's, many associated with seemingly valid domains who's servers are probably infected (including on fully updated servers I manage). Yikes! Glad I did all this a while back. – jdmayfield Jun 22 '19 at 02:58
  • 1
    It is worth mentioning that this exploit was successfully used to [inject a cryptojacking miner](https://serverfault.com/questions/976983/) on a server of mine. Checking for a `seasame` file under `/tmp/`, `/var/tmp/` or `/dev/shm/` might be a good idea. – Skippy le Grand Gourou Jul 28 '19 at 22:24
  • 1
    @skippy Thank you for that! There are so many people who would never admit something like this. You require a thumbs up! How did you find it? Were there any signs, like unusually excessive CPU use or memory consumption, or anything else that gave you a clue? Or was it niced down very low so as to be harder to detect? Is there a particular process you that seems to be associated that can be looked up via ps or top etc? This is great stuff. Can you confirm if this happened before or after this hack and/or upgrading? Thank you again for the info. I am sure this will be helpful for some. – jdmayfield Jul 29 '19 at 10:17
  • 1
    By chance, I happened to check the crontab after reading a question about compromised crontab… The process (disguised as /boot/vmlinuz) is not niced down at all, it actually kills all legit process using more than 30 % CPU in order to keep all the CPU to itself. This happened the day before I upgraded Exim. – Skippy le Grand Gourou Jul 29 '19 at 16:12
  • Wow! It disguised itself as a kernel? That sounds not nice in more ways than one! Did you have to rebuild the system? Which distro? I lucked out, in a sense-- the frozen messages and alerts from the mail daemon started showing up less than 48 hours after I upgraded to the patched version. Still, this made me a little paranoid; why I banned incoming addys with dollar-signs and curly brackets. RFC-legal or not, I don't recall ever running across email addresses with any of those, and to my knowledge none of the big guys allow them either. Better safe than sorry. – jdmayfield Jul 30 '19 at 08:28
  • I have to add, part of my reasoning here is simply that nobody really knows there isn't a similar exploit possible. If one is found in the future, and the characters are rejected before the possibility of string expansion, it would prevent it from happening even if someone does find a similar exploit. – jdmayfield Jul 30 '19 at 08:36
  • Debian, and I guess I *should* rebuild the system, although removing the malware files and restoring the crontab and modified/deleted files *might* suffice, along with changing the root password just in case — but who knows if the main program didn't install some backdoor ? – Skippy le Grand Gourou Jul 30 '19 at 09:01
  • Also, it's worth mentioning that I received a [`maldet`](https://en.wikipedia.org/wiki/Linux_Malware_Detect) alert when manually downloading the malware for tests — too bad I didn't receive such an alert before, not sure why. – Skippy le Grand Gourou Jul 30 '19 at 09:02

2 Answers2

3

This is what we have at work for people that haven't updated exim. It's more specific to match ${run{...}} string. Also a script here to locally check if your server is vulnerable, exim-cve-2019-10149

If acl_smtp_rcpt is not already set to acl_check_rcpt (possibly via MAIN_ACL_CHECK_RCPT), change

acl_smtp_rcpt = acl_smtp_rcpt

to

acl_smtp_rcpt = acl_check_rcpt

After begin acl, or in /etc/exim4/conf.d/acl/30_exim4-config_check_rcpt in split configuration, add or modify :

acl_check_rcpt:

  deny
    message = Restricted characters in address
    domains = +local_domains
    local_parts = ^[.] : ^.*[@%!/|] : ^.*\N\${run{\N.*}}

  deny
    message = Restricted characters in address
    domains = !+local_domains
    local_parts = ^[./|] : ^.*[@%!] : ^.*/\\.\\./ : ^.*\N\${run{\N.*}}

  accept
  • Nice test! The more s[ecific regex is a nice touch as well. For some it may be required. Personally, I am wondering about similar expansion exploits that may not include the run directive, so still would want to at least keep it no less specific than to '${'. Thanks for the additions! Especially the test. – jdmayfield Jun 22 '19 at 02:56
  • @jdmayfield this obviously is just a temporary fix, we're making everyone to update instead of relying on the conf. And since the run expansion is part of exim (can't turn it off that I know of), we're rather not mess with it. – Andréw Hüang Jun 23 '19 at 03:15
  • I couldn't agree more. – jdmayfield Jun 23 '19 at 05:04
1

Quick version:

To protect your server from attacks from the CVE-2019-10149 "Return of the WIZard" exploit, whether on patched or unpatched Exim versions, if you are on a Debian-based OS with Exim "split-config", add the following lines to your 00_local_macros file in /etc/exim4/conf.d/main folder, or whatever path that file is in on your system:

CHECK_RCPT_LOCAL_LOCALPARTS  = ^[.] : ^.*[\${}@%!/|`#&?] : ^.*/\\.\\./
CHECK_RCPT_REMOTE_LOCALPARTS = ^[.] : ^.*[\${}@%!/|`#&?] : ^.*/\\.\\./

In addition, it is a good idea to add:

smtp_banner = $smtp_active_hostname ESMTP $tod_full      

This will change your Exim servers greeting to disclude it's version number, as well as the fact that it is Exim, so service scanners like https://shodan.io won't show that information (some-time later) as a surface attack vector specific to this exploit (or any others based on Exim).

Other systems may vary slightly, but basically you simply deny recipent addresses on your system that contain the additional character '$', '{', and '}'. You could just add the dollar-sign to your config where the illegal characters are designated, but I prefer to be safer in the event a similar exploit is found, so I added the curly-brackets as well.

The only caveat is if you have addresses on your local system that contain dollar-signs or curly-brackets, they won't get mail. But who has $$bling{me}@whatever.com for an address? It won't block sending out mail to those addresses, however, in the unlikely event you sending to a my{$$$}@somewhere-else.com

This has the added advantage of not only rejecting the exploitive mail, but sending a bounce message about it to the offending server.

I explicate on this subject here:

https://www.bleepingcomputer.com/forums/t/699962/mail-remote-code-execution-attempt/#entry4818756

Previously on this bat-channel---

Background: This is in reference to a patched Exim (which may be < 4.92, but likely > 4.91). First, upgrade your Exim, sshd, and really your whole system, as high as you can reasonably and safely do.

The patched version 4.90_1 indeed seems to have prevented break-ins, as I have not found any of the known evidence of the "Return of the WIZard" Exim worm outlined in the articles I have read. In addition, Exim on Ubuntu (and I would think other Debian-based flavors) run under user Debian-exim, not as root, so this in itself may have prevented a lot of damage. As bad as a compromise of the Debian-exim user may be, it's a lot better than root. In any case, I haven't seen any signs of successful tampering.

This was still alarming, however. The one thing I was seeing were periodic references to frozen messages in my Exim mainlog with RCPT's all starting with ${run{ followed by coding that would reduce down to things like /bin/bash and a series of commands, including the scary wget to some shady endpoints.

Most specifically, they all seemed to come from 89.248.171.57 --an IP that resolves to scanner20.openportstats.com . If you are seeing similar frozen messages, you can go to openportstats.com and look up your IP to see if you're listed there. The site appears a little like shodan.io .

Question is, are they doing this on purpose? Or are they infected? I initially thought the former, but not totally sure as I would've thought there would be some mention of results regarding the Exim vulnerability when I looked up my IP, but there was only basic info on ports and data extracted from initial connections.

Anyway, moving on.

The (rather vague) article from Ubuntu regarding the patched Exim versions is: https://usn.ubuntu.com/4010-1/

Here is a more in-depth article that delineates some of the actual mechanics involved: https://hackernews.blog/exim-4-87/#more

Personally, I think many sources I keeping their lips tight about this, but the paranoia is doing more harm than good, people! The cat's out of the bag for bad actors. Those trying to fight it, and/or just protect themselves, need more information as it comes.

This fix is intended to be in addition to updating your Exim to a patched version.

I do not know if this would protect an un-patched version from this exploit, and hence the worm. But it seems likely based on what I have read. If for some insane reason you cannot update your Exim, definitely give this a try and let us know how it works out.

I do think the vulnerability is in the routers, not the ACL's, but I've scanned so much garbage on this my memory's a little fried right now.

What I can say, because I've tested, is that this will reject any mails with a RCPT address containing a dollar sign. That include the ${run's.

I found the long answer here, regarding rejecting the ${run's. https://marius.bloggt-in-braunschweig.de

Seems obvi in retrospect, but then I was over-thinking it.

HERE IT IS:

You can test the following fix live --BEFORE YOU DO THE FIX-- by opening an additional terminal and running

tail -Fn +0 /var/log/exim4/mainlog | grep "{run"

Then, send from a local mail account to ${run{true}}@yourdomain.name

And send from a remote address to that same address.

Then do the fix below.

Note: exchange the log path for your mainlog path, if different.

Anyway, I have the Debian split-file config, so I just found the lines

 90 .ifndef CHECK_RCPT_LOCAL_LOCALPARTS                                                                                                                                                     
 91 CHECK_RCPT_LOCAL_LOCALPARTS = ^[.] : ^.*[@%!/|`#&?]                                                                                                                                   
 92 .endif                                                                                                                                                                                  
 93                                                                                                                                                                                         
 94 .ifndef CHECK_RCPT_REMOTE_LOCALPARTS                                                                                                                                                    
 95 CHECK_RCPT_REMOTE_LOCALPARTS = ^[./|] : ^.*[@%!`#&?] : ^.*/\\.\\./                                                                                                                    
 96 .endif

and changed them to:

 90 .ifndef CHECK_RCPT_LOCAL_LOCALPARTS                                                                                                                                                     
 91 CHECK_RCPT_LOCAL_LOCALPARTS = ^[.] : ^.*[\$@%!/|`#&?]                                                                                                                                   
 92 .endif                                                                                                                                                                                  
 93                                                                                                                                                                                         
 94 .ifndef CHECK_RCPT_REMOTE_LOCALPARTS                                                                                                                                                    
 95 CHECK_RCPT_REMOTE_LOCALPARTS = ^[./|] : ^.*[\$@%!`#&?] : ^.*/\\.\\./                                                                                                                    
 96 .endif

Note the addition of the "\$" in both.

Then restart Exim.

Test for successful rejection, with the same method:

Send from a local mail account to ${run{true}}@yourdomain.name

And send from a remote address to that same address.

If successful, and normally configured for bounces, you should immediately get a bounce for the second pair of send.

Look at the terminal you opened with the tail of the logs. You should see rejects on the second pair.

OTHER IDEAS ON TESTING

So basically the vulnerabilty simply executes shell commands, just like you would at a terminal. You just send your commands to/in an address like this: ${run{my_command}}@mytargetdomain.com . There is a small catch, typically illegal characters (for an email address) have to be encoded. You could form your script this way, or you could drop a script in your /bin directory that does something, say-- save a file to your root directory.

If you send it from a local address and it appears, you'd better fix it quick, because that means it's run as root. You might also try the same, but make a folder i.e. /runtest owned by the user your Exim runs as (On Ubuntu typically user Debian-exim). If it shows up, you're still in shady territory, but the worst of the worm variants seem to need root (so far that I've seen publicized, but that could change any moment). Still fix it.

If your wondering why I used ${run{true}}@mydomain.com, it's because I was looking to test the reject in a safe manner that won't blow up my server.

If anyone can elaborate on any of this, please, please do. The world needs to hear it.

I am really surprised that not a single person has put in a peep, good, bad, or indifferent.

I will update this periodically as I have time, and information comes in.

jdmayfield
  • 271
  • 2
  • 11
  • Seriously guys (gals?). I've never gone this long without a some kind of comment or vote, good or bad. Anyway, rejecting mails with a '$' in the RCPT will prevent the whole problem, even on unpatched servers, so long as check rcpt and check remote rcpt are used in the ACL's (normally by default). If uour Exim rejects it before 'delivery' you should be ok. According to the docs I linked to above, it is at the delivery stage that the vulnerability exists (i.e. the recipient string only gets "expanded" then, not before). Can someone comment on this... sheesh. – jdmayfield Jun 16 '19 at 06:50
  • Removing addresses solely on the dollar sign criterion is not a good solution, since [this is a valid email character](https://stackoverflow.com/questions/2049502/what-characters-are-allowed-in-an-email-address). – Skippy le Grand Gourou Jun 16 '19 at 21:51
  • You could always unallow '${'. Although $ may be an allowed character according to specs, most larger companies don't seem to allow it in their addresses. I'd rather be safe first. Can always change it if it becomes an issue. Honestly, the only addresses I've seen with a dollar sign were from spammers anyway. – jdmayfield Jun 17 '19 at 03:59
  • @Skippy thanks for commenting though! All ideas are welcome. – jdmayfield Jun 17 '19 at 04:00
  • Or maybe just unallow '{'. That should work too, I would think. – jdmayfield Jun 17 '19 at 04:02
  • This is awesome. Although I had been seeing many log entries from various sites and IP's (many from the typical bot-net attack surfaces, but also many from apparently legitimate sites, probably infected), since banning '$'s in RCPT addresses (I also added '{' and '}' to the list), and therefor bouncing messages with a clear indicator as to why and what the recipient address was, I saw a marked and steady decrease in attacks, finally ending. The last ${run attempt was on 2019-06-20 04:51 UTC, less than a week after implementing this. – jdmayfield Jun 29 '19 at 08:56
  • Some of this may be due to fail2ban picking up on the excessive rejects, but I think this has been also beneficial for admin (and users) of those attacking servers that were legit, as they would have been clearly notified was is happening, and that something was wrong. – jdmayfield Jun 29 '19 at 09:02
  • **Note: To be clear-- this doesn't ban using those characters in addresses I'm sending to, only destination addresses for incoming mail. There are no addresses on my systems that have dollar-signs or curly-brackets in them, nor will there be, so this is totally safe regarding delivery. In the event someone needs to communicate to someone on another server who does have those characters in their address, they are allowed. – jdmayfield Jun 29 '19 at 09:05
  • As an aside, I also changed my server's smtp issue statement, removing references to both version, and even the fact it is Exim. Checking various scanner sites such as https://shodan.io reflected the change, and this may also have reduced the attack vector. So far no problems, and no complaints, no apparent issues. All seems to be operating normally. – jdmayfield Jun 29 '19 at 09:11