6

My LAMP server has been hit with some kind of crypto mining malware. Crontab is clear and unused, Clam can't seem to detect anything.

It appears to automatically create these files in my /tmp folder at random times of the day.

  • phpIij8W8.c
  • phpIij8W8_fkk2qr2rqjikiewz (Always has a random name after I delete)

Inside of this phpIij8W8_fkk2qr2rqjikiewz:

threads = 1
mine = stratum+tcp://44XNuUyCFUgjG23yPfCHp(cut for content...etc):x@monerohash.com:3333/xmr

They all begin with "php" so I'm assuming some kind of PHP script is on the server that is automatically creating these files in the tmp folder and running them.

I've attempted to use iNotify to determine what/how these have been created and all I've been able to determine is this:

File phpvlnebP_fkk2qr2rqjikiewz has been just created by MOVED_TO

Can anyone give me some tips to narrow down the script that is creating these random files? Or if there's some way to get iNotify to give me more information when the file is created/accessed. Or heck, if there's a better tool our there to diagnose this problem.

Ubuntu 16.04.3

Taurian
  • 163
  • 1
  • 5
  • 1
    Do you have samba ports open on the machine? If so that was the likely infection vector. If you don't need it disable the samba service and firewall off 445 and 139. I'd also suggest firewalling off monerohash.com and destination port 3333. Have you checked for running processes with obvious names like "miner*"? – Hector Oct 25 '17 at 11:47
  • Can you run netstat -l and let us know the output? And if you suspect its php related have you looked through all active apache sites/conf/mods etc? – Hector Oct 25 '17 at 11:53
  • Thanks for your response Hector. I had to disable UFW temporarily as it was creating insanely large log files. Samba is: inactive (dead). I've run: grep -r on "Miner", Stratum and a few other keywords. I was hoping there was a way to determine how those php* files ended up in the /tmp folder. Everytime I delete them, they simply come on back. – Taurian Oct 25 '17 at 13:24
  • @Taurian what is with an infected Web Application (maybe even PHP based). That way the Attacker must only send an Request to create those files. Or better (? worser) he just do that on every request that is incoming. – Serverfrog Oct 25 '17 at 13:33
  • @Taurian - there are a large number of ways.We don't even know how you were infected - it could even be reinfection. Turning off the firewall has just made the problem a lot worse. Why not just disable logging? ("sudo ufw logging off"). Have you checked "ps aux " for unknown or dodgy looking running processes? And have you checked the apache config? – Hector Oct 25 '17 at 14:03
  • Other log files get filled up. It isn’t limited to UFW. I’d rather have the infection continue communicating with its C&C so I can learn more about it. My question was about any tools besides INotify or even if INotify can provide more data about how/who is creating those php* files in the /tmp folder. I’ll alter my question to be a little more clear. – Taurian Oct 25 '17 at 17:38
  • 1
    Also check my other answer, here https://security.stackexchange.com/questions/172396/some-bot-keeps-posting-this-to-my-server/172571#172571 – Aleksandar Pavić Oct 31 '17 at 17:30
  • Aleksandar strikes again! – Taurian Nov 01 '17 at 00:33

3 Answers3

2

I had same problem, and I've narrowed down, hackers were able to intrude in some old and non-upgraded, wordpress.

Probably the best and fastest way to see who is consuming how much time on your Ubuntu 16.04.3 server is to install htop

sudo apt install htop

then type sudo htop

It will show you under which username is eating how much CPU enter image description here

And if you identify some process that's eating much cpu, you can check it by lsof -p <pid>

lsof stands for list open files, to see it's full set of commands, type man lsof

However, it all depends upon how is your PHP being executed, and what hackers actually have done to your system.

Another good way to see, what is Apache exactly doing is to enable mod_status

Usually, on newer ubuntu's it's:

sudo a2enmod status

And after that add this to your 000-default.conf website:

<Location /server-status>
    SetHandler server-status
    Require ip 127.0.0.1
    Require ip ::1
    Require ip X.X.X.X
</Location>

Replace X with your actual IP...

Or you can access it with lync for example from your server's console like

lynx 127.0.0.1/server-status

And the output should look like: enter image description here

On my other post https://security.stackexchange.com/questions/172396/some-bot-keeps-posting-this-to-my-server/172571 you can see how to improve security of your server and prevent such attacks.

1

To be clear, your best bet here is to nuke it from orbit. You presumably have a copy of your application somewhere, and also a copy of your database. If you really want to try to figure it out, then you should probably start from the perspective that you have one or two intrusion vectors:

  1. Your web application
  2. Your system

While I'm guessing wildly, it seems that it is a safe bet that the PHP file you keep deleting isn't the actual malware, but rather a by-product of the malware. I know you know this, but to be clear: you haven't done anything to remove the malware, just the symptoms of it. Discovering and removing the malware itself will vary wildly depending on whether or not it is in your web application or in your system. Who is the owner of the file you keep deleting? If it is the user your webserver runs on, then the infection probably lies with your web application. If so, here's some details that might help:

Web Application Intrusion

  1. You know when the payload is being created in your temporary directory. See if there are any suspicious looking requests in your HTTP logs that correlate with that time period. If the intrusion vector is your web application, then there are likely calls coming in through your HTTP server from a C&C server that is triggering the payload execution.
  2. Immediately update your software application to the latest version, presuming that you are deploying third-party software (wordpress, etc...)
  3. Does your web user have write access to the folder where the web application lives? If so, you're SOL. If they got in through your web application, this means that they have found a remote-code execution vulnerability. With write access to your application directory, that means that they could have installed backdoors anywhere they wanted to inside your application. Unless you have old backups and can determine for sure that nothing malicious was added to your code base (which is a big pain to do), your best bet is to NUKE IT FROM ORBIT
  4. Conversely, it is possible that you have an open vulnerability which isn't patched, and your system itself is unmodified (more likely if your web server doesn't have write access to the web application). In this case the payload keeps returning because the C&C server keeps re-exploiting the security vulnerability. In that sense the problem will go away once you get your security problems patched. Make sure and upgrade all parts of your web application. If this is a home-grown application then you may very well be SOL: unless you can find and close your own vulnerabilities, nuking it from orbit may not stop them from getting back in. In this case you will hopefully get some help from your server logs. If you can find evidence of the initial intrusion in your logs, that will help you find and patch the vulnerability.

System Intrusion

This one is probably less likely, but it is possible your system itself was compromised (insecure service, crackable SSH passwords, etc...). Short of lots of digging through logs, it may be hard to determine for sure if it is your system itself that was the weak point. It will probably be much easier to determine that your web application is the problem (from my own experience, it probably is your web application that is the culprit anyway). Either way, if you can't determine for sure that it was your web application, your system is the only other culprit.

The answer

Either way, the trouble is that it is going to be very hard to determine for sure where the problem got started, and harder still to tell whether or not you got them out. Even if you manage to get the cryptominer to stop putting itself in place, what guarantees do you have that there isn't a more silent and more malicious process hiding elsewhere? Background process that scrapes your database and sends it off to a remote C&C server once a week? That's a great way to have the personal information of anyone who logs into your system get leaked. Not to mention the threat to your own passwords and account credentials. This is why, regardless of where the intrusion started, there is only one answer:

  1. NUKE IT FROM ORBIT
  2. Spin up a new server following all best practices (minimum services, proper firewall, everything fully patched)
  3. Use all new passwords for all services.
  4. Find a copy of your application that is KNOWN TO BE CLEAN. If using third party software, make sure and fully upgrade/patch.
  5. Restore from the latest copy of your database. Make sure there aren't any extra "users" with administrator access, and change all administrator passwords
  6. Make sure the new database has no triggers/procedures that aren't from you.
  7. If this application is a home-grown system then it is time to invest heavily in security training and re-writes, unless you know for sure that your system was not at fault.
Conor Mancone
  • 29,899
  • 13
  • 91
  • 96
  • @Taurian I realize this isn't the answer you are looking for, but still: nuke it from orbit. Is the file owned by your web application user? – Conor Mancone Oct 25 '17 at 17:43
  • @Taurian No guarantees of course (especially from here), but I would strongly suspect that the culprit is a vulnerability in your PHP application. In that case, the service creating the file in your /tmp folder will be your web application (apache or php-fpm, most likely). It is possible that you do not have a phantom process running anywhere, unless the malware made a job from your web application to your system. I would start digging through your access logs around the time of file creation: that will likely get you farther than anything else. – Conor Mancone Oct 25 '17 at 18:43
  • nice answer.. reinstall everything, change everything, do everything from scratch.. that's not a solution. find the problem and eliminate that's the solution the question is more: where to start – iRaS Jan 29 '18 at 06:27
0

One of my servers somehow managed to catch it at one datacentre, but 2 of my other servers running the same software, each hosted with a different provider, did not get the infection - a nasty one at that, causing the CPU (quad core) to max out.

After searching for an answer I came across this board and although I like the NUKE IT OUT OF ORBIT approach, I may have found a solution after trying many approaches for hours.

Here's the synopses:

I figured if I wanted to stop the existing processes from firing up, I need to obviously delete the executables - not exactly. Since we're dealing with an infection, the processes kept coming back after a few milliseconds. After I deleted the executables, they came back - I managed to catch an SSH connection going out to the C&C server, will post it here later once I log it on my experimental machine. Next step I did to prevent it from launching, was to find all executables and instead of deleting them, I set their permission to nobody:nobody and 444 (I wanted to still review the contents you can set it to lower). What this did is occupy the name of the file used by the rogue program so that the SSH connection could not replace the contents of the file, and also prevented it from executing rendering it useless. This seems to work for stopping the processes from automatically starting and allowing Remote Code Execution and effectively quarantining the executables .

Please note this is only a patch solution and not a remedy to fix the problem. I will be reviewing the code commit by commit (in git) and eyeballing log files for a while until I figure out how the payload got in there in the first place.

CentOS7

Steps:

install htop:
#: sudo yum install htop

#: htop

Sort by CPU% and note the name of the process consuming 99% or 100%

find the executable:

#: sudo find / -type f -iname "[FILENAME]"

replace [FILENAME] with the name you grabbed from htop

#: sudo chown nobody:nobody /path/to/[FILENAME]
#: sudo chmod 0444 /path/to/[FILENAME]

Shut down the currently running executables

#: sudo htop

find the process, select and hit the "K" key to kill the process. At this point, it should not come back. If a new process popped back up, repeat the steps above.

Maybe someone will have time to write a sh script to do it automatically.

[UPDATE:] ALso watch out for CRON jobs for user apache - runs every 15 mins to download the payload.

Shib
  • 1
  • 2
  • While it's nice to be able to stop a process that's using too many resources, and you did mention this already, I just want to reiterate this for people just scrolling through: your suggestion does *absolutely nothing* to the root cause of the problem. You are likely *still* vulnerable to whichever `file inclusion` or `remote code execution` caused the problem in the first place. – Mark Buffalo May 12 '18 at 02:28