0

Hello everyone and hopefully somebody can give me a first step where I can begin investigating the reason to know why our Linux server appears to have attacked our service provider where the server is hosted on.

Today I received e-mail that one of our servers has been compromising one of our service provider customers as part of a coordinated DDoS botnet so they had to null-route it in order to mitigate so it went down and more or less all our business was down. They reviewed the captures from this attack and do not believe that our IP address was spoofed based on the limited number of distinct hosts attacking to they.

This is our Linux server hosting many different services connected to other our internal servers in infrastructure.

I need to investigate and be able to observe the attack that likely saturated the network adapter of the source. Since the source device is a member of a botnet that is being used for many attacks and I should see many other mysterious bursts of outbound traffic BUT THE PROBLEM IS:

We don't have any monitoring on this server installed so I cannot monitor the traffic that was going out from the server so the question is:

Is it possible to somehow track outbound traffic that was attacking our service provider's customers in Linux? Any commands that could help me? Maybe there are logs recorded?

I do have an information about last timestamps (at the very left). Source and destination IP addresses, protocols, and ports. Unfortunately I don't know where to start as this server doesn't have any monitoring and I don't have great Linux knowledge as I am pretty desperate now and of course, everything need to happen right before Christmas.

Any information will be kindly appreciated.

EDIT: I used journalctl by a given timestamp and now i'm able to see that there are numerous of attempts trying to connect to ssh but unsuccessfully tho one time session opened for root user with given response:

CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)

After that, session for this user is closed.

Anyone knows what this should mean about?

sintezators
  • 1
  • 2

2 Answers2

0

In general, network bandwidth usage isn't logged or monitored on Linux servers by default. If you didn't install a monitoring system before than you don't have that data.

AlexD
  • 8,179
  • 2
  • 28
  • 38
  • what you think about `journalctl`? I used it by a given timestamp and now i'm able to see that there are numerous of attempts trying to connect to ssh but unsuccessfully tho one time session opened for root user with given response: `CMD ( [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean)` After that, session for this user is closed. – sintezators Dec 23 '21 at 18:12
  • The messages are just normal log noise. If you are not a forensic expert, you won't find anything. You need to go and read https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server – AlexD Dec 23 '21 at 18:26
0

If you were not logging at the time, you will not have details. Certainly not at the level of evidence that your host sent certain flows. In the future, consider enabling monitoring and logging, such as enabling logging of new connections via a firewall.

Backup the misbehaving host in case it you will have forensics done on it. Do not give (copies of) this host network access, and never give it internet again. It is likely compromised.

Destroy and rebuild the host from known good sources. Such as install a fresh copy of the OS. Restore data from backup.

As a follow up, get help with doing a detailed root cause investigation on how you might have been sending malicious traffic. Successful ssh logins, evidence of malware, installed software missing security patches, review the flows described by your hosting provider. We cannot help with the details in this Q&A format, however.

John Mahowald
  • 30,009
  • 1
  • 17
  • 32