1

We have a live AWS EC2 instance with Ubuntu 14.04, which I have been handed to maintain. An oddity that I have found is that I have a process of /tmp/systemd using all available CPU:

top - 11:35:20 up  2:34,  1 user,  load average: 1.13, 1.16, 1.15
Tasks: 114 total,   2 running, 111 sleeping,   0 stopped,   1 zombie
%Cpu(s): 32.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 67.3 st
KiB Mem:   2048516 total,   726492 used,  1322024 free,    38784 buffers
KiB Swap:        0 total,        0 used,        0 free.   439788 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 1797 www-data  20   0  192900   4008   1140 S 95.2  0.2 107:42.82 /tmp/systemd
 1145 mysql     20   0  681596 121328   7560 S  3.6  5.9   2:35.84 /usr/sbin/mysqld
 3669 ubuntu    20   0  103084   1904    920 S  0.7  0.1   0:00.06 sshd: ubuntu@pts/0
   40 root      rt   0       0      0      0 S  0.3  0.0   0:04.96 [watchdog/0]
 3692 ubuntu    20   0   23728   1652   1104 R  0.3  0.1   0:00.11 top
    1 root      20   0   33556   2880   1480 S  0.0  0.1   0:03.20 /sbin/init
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 [kthreadd]

I thought the /tmp/systemd seemed a tad odd for a command source, so I checked the /tmp directory - and it's completely empty.

Running an strace seems to indicate that it is polling for something to finish:

ubuntu@ip-10-0-0-157:~$ sudo strace -fvvp 1797
Process 1797 attached with 6 threads
[pid  1801] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1802] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1803] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1800] futex(0x799404, FUTEX_WAIT_PRIVATE, 4, NULL <unfinished ...>
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 290556741}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] clock_gettime(CLOCK_REALTIME, {1524656311, 141773213}) = 0
[pid  1797] <... epoll_wait resumed> {}, 1024, 27) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 328709201}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 328773526}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1797] <... epoll_wait resumed> {}, 1024, 500) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 839857928}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 839934892}) = 0
[pid  1797] epoll_wait(7, {}, 1024, 18) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 860012749}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9448, 860081346}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1797] <... epoll_wait resumed> {}, 1024, 479) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 350734613}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 350806967}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0
q[pid  1797] <... epoll_wait resumed> {}, 1024, 500) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 905946457}) = 0
[pid  1797] clock_gettime(CLOCK_MONOTONIC, {9449, 906007520}) = 0
[pid  1797] epoll_wait(7,  <unfinished ...>
[pid  1799] sched_yield()               = 0
[pid  1799] sched_yield()               = 0

I've tried updating and rebooting the server, but it changes nothing. I can kill the process, but approximately 1 hour later it will restart.

Can you please advise on how to locate source of this or how I can find out what it is doing and what is starting it?

Zoek UK
  • 13
  • 2
  • 3
    There is no /tmp/systemd binary or temporary binary, given that the user which started the process is non-interactive "www-data" I would suggest that your webserver was compromised. Check for any suspicions outbound connections, re-mount /tmp with noexec and check if everything owned by www-data is as it should be. Looks like Tiny XMR mooner - https://xorl.wordpress.com/2017/12/21/the-tiny-xml-mooner-linux-cryptominer-malware/ – Daniel Apr 25 '18 at 13:20
  • @Daniel this answer seems spot on. Should be an answer and not a comment. – Mark Stosberg Apr 25 '18 at 13:46
  • Thank you - and yes, if you create this as an answer, I'll mark it complete - thanks again. I've remounted /tmp with noexec, nodev and nosuid, and I'm looking to see how they have gained entry into the server. – Zoek UK Apr 25 '18 at 14:11

1 Answers1

1

There is no /tmp/systemd binary or temporary binary, given that the user which started the process is non-interactive "www-data" I would suggest that your webserver was compromised.

Check for any suspicions outbound connections, re-mount /tmp with noexec and check if everything owned by www-data is as it should be. Look for shell scripts or other executables that should not belong to www-data.

Looks like Tiny XMR mooner - https://xorl.wordpress.com/2017/12/21/the-tiny-xml-mooner-linux-cryptominer-malware/

Best practice is such cases is to make image of the machine (with dd for example) and analyze it offline in environment with no Internet access.

Daniel
  • 302
  • 1
  • 5
  • Best practice is to take the machine offline immediately, analyze it, and then destroy it. You can never be sure you have fully cleaned a compromised server, and there is little to be gained by trying. https://serverfault.com/q/218005/153161 – Michael - sqlbot Apr 26 '18 at 00:39