Some program is making connections to an unknown subnet, but couldn't find the process that initiates the connection

Question

We have a server that is making a lot of outbound connections to a particular subnet, I am not able to find out what process is making it.

I could see the connections in tcdump -

[root@something ~]# tcpdump -i ens192 -v | grep 26379
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
    hosname.sometohing.43494 > 172.31.20.63.26379: Flags [S], cksum 0xe734 (incorrect -> 0xe6c9), seq 258461406, win 26720, options [mss 1336,sackOK,TS val 1947447040 ecr 0,nop,wscale 7], length 0
    hosname.sometohing.43494 > 172.31.20.63.26379: Flags [S], cksum 0xe734 (incorrect -> 0xe2dd), seq 258461406, win 26720, options [mss 1336,sackOK,TS val 1947448044 ecr 0,nop,wscale 7], length 0
    hosname.sometohing.37346 > 172.31.42.22.26379: Flags [S], cksum 0xfd0b (incorrect -> 0x5843), seq 1954738294, win 26720, options [mss 1336,sackOK,TS val 1947449600 ecr 0,nop,wscale 7], length 0
    hosname.sometohing.37454 > 172.31.42.22.26379: Flags [S], cksum 0xfd0b (incorrect -> 0xe93e), seq 1610576711, win 26720, options [mss 1336,sackOK,TS val 1947449675 ecr 0,nop,wscale 7], length 0
    hosname.sometohing.43494 > 172.31.20.63.26379: Flags [S], cksum 0xe734 (incorrect -> 0xdb09), seq 258461406, win 26720, options [mss 1336,sackOK,TS val 1947450048 ecr 0,nop,wscale 7], length 0

I am trying the lsof to see if finds something, but nothing for 2 minutes, but tcpdump keeps printing that host is sending traffic.

while true; do lsof -i | grep 172.31;done

And then netstat with all possible switches. antp, vunp, vutp, e, o and everything I could think of. I don't see any output.

I would be really grateful for help.

Possible duplicate of [How do I deal with a compromised server?](https://serverfault.com/questions/218005/how-do-i-deal-with-a-compromised-server) — Tero Kilkanen, Nov 16 '19 at 10:42
How often do these SYN packets arrive? What is the configuration of `ens192` interface? — Tero Kilkanen, Nov 16 '19 at 10:44

score 2 · Accepted Answer · answered Nov 16 '19 at 12:39

2

Install bpf-tools and run tcpconnect. Every connect() will be traced to get its PID and COMM, which is actually lightweight compared to packet capture.

Also take a packet capture, run it through Wireshark, and see what protocols are revealed with the dissectors.

answered Nov 16 '19 at 12:39

John Mahowald

30,009
1
17
32

I will try this and comment the outcome here, thank you so much. – Vignesh SP Nov 16 '19 at 23:38

score 0 · Answer 2 · answered Nov 16 '19 at 06:04

I have not had this issue before, but I expect netstat will assist you. Try running

  netstat -tup

This will show you a list of tcp and udp connections. The last column will hopefully show you a PID / Program Name to allow you to identify the associated process.

Some program is making connections to an unknown subnet, but couldn't find the process that initiates the connection

2 Answers2