0

We recently had some servers hacked (Ubuntu, various flavours) which installed a Monero miner that starts a process called watchbog. We scrubbed them clean and blocked access to where they can update themselves from but on reboot the servers start to rebuild the miners installation (and fails)

It is creating a directory /tmp/systemd-private-d3883bec41f94ab0b3d927e3022873b1-systemd-timesyncd.service-jVvrE0 and some subdirectories and then stops. The random bits are random each time

What I want to know is what is rebuilding this on boot. There does not seem to be anything obvious in the logs or the various boot scripts. Where else can I look?

Peter Hickman
  • 1
  • 1
  • I have gone over all that but other than being rather handwavy they do not address my needs. We can and do burn the machines down if they are infected, which is why we use Chef. But this will just rebuild the server as it was and then it is just a matter of time before it happens again. I need to know how the hack is reviving itself so I can update the installation to address it – Peter Hickman Feb 28 '19 at 10:56
  • Your _completely fresh_ installed machines are getting infected again? – Lenniey Feb 28 '19 at 11:00
  • This is a clone of the original infected machine. The original was rebuilt but as I say it probably has the same vulnerability. I need to know more to protect the new server – Peter Hickman Feb 28 '19 at 11:08
  • That's covered in the linked post. You start from scratch. Otherwise you can't _ever_ be sure. – Lenniey Feb 28 '19 at 11:11
  • Thats the problem. If I start from scratch I end up with a server that is configured exactly the same way as it was when it was infected. Which means it will get infected again. I need to know more to build better defences, knowing how it is rebuilding itself on boot will allow me to protect the new server better or at least be better at detecting the infection – Peter Hickman Feb 28 '19 at 11:15
  • No I mean from **real** scratch. No Chef deployment or anything. Have a look [at this question](https://serverfault.com/questions/909397/ubuntu-14-04-tmp-systemd-process-using-100-cpu), looks like it's the same miner. – Lenniey Feb 28 '19 at 12:20
  • We use Chef to build our servers from scratch. i.e. provision a new vm and build that – Peter Hickman Feb 28 '19 at 13:02
  • The malware got in through a vulnerability in some web app. It might have elevated privileges through some other local vulnerability. Keep your systems up to date. Internet facing systems are strong candidates for _automatic_ updates, and updating your web applications probably should be automated too. – Michael Hampton Feb 28 '19 at 13:30

0 Answers0