How to find out PID of the process sending packets (generating network traffic)?

Question

Couple of weeks ago I had an issue where I changed DNS addresses in large network of around 300 nodes. After that, some of the nodes still continued to ask old DNS servers, although resolv.conf was ok, and host/nslookup were querying new DNS servers.

Looking at tcpdump and trying to record requests with iptables logging, I confirmed that indeed some of the hosts were still sending queries to old nameservers.

I took one of the hosts out of production and started shutting down services / stracing processes in an attempt to find out the culprit.

At the end - it was lldpd daemon, which obviously cached nameservers at startup and didn't even notice changes in resolv.conf.

So, my question is - is there a more intelligent way to find out which PId is generating specific kind of traffic? I tried with auditctl but without much success. CentOS 6 is in question but if there is solution for any Linux distro, I would appreciate it.

Answer 1

What's wrong with the auditctl?

You would do it like this

1) Define your audit rule to audit sendmsg and sendto system calls. These system calls are used during name resolution.

auditctl -a exit,always -F arch=b64 -S sendmsg -S sendto -k send

2) Now search for your audit records. You can grep based on the remote DNS IP here

ausearch -k send -i|grep -A2 "serv:53"

In the below example you can see that application which was responsible for the systemcall is called dig

ausearch -k send -i|grep -A2 "serv:53"
type=SOCKADDR msg=audit(10/31/2016 15:24:56.264:176998) : saddr=inet host:172.16.0.23 serv:53 
type=SYSCALL msg=audit(10/31/2016 15:24:56.264:176998) : arch=x86_64 syscall=sendmsg success=yes exit=29 a0=14 a1=7fa1919f9ac0 a2=0 a3=7fa1919f9780 items=0 ppid=31729 pid=32047 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts5 ses=52 comm=dig exe=/usr/bin/dig subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=send


comm=dig exe=/usr/bin/dig

And the way to differentiate to which remote DNS request is send is here. So you would just have to grep for a particular DNS host.

saddr=inet host:172.16.0.23 serv:53 

Or even better - see what DNS hosts are used (I have only one in this example)

ausearch -k send -i|grep "serv:53"|awk '{print $6}'|sort|uniq -c
      3 host:172.16.0.23

And then narrow down which apps are using those particular hosts.

Edit 1: Actually I just did strace of a simple ping to a host. Seems like sendmsg is not always used. Here is what I see

socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.16.0.23")}, 16) = 0
gettimeofday({1477929832, 712018}, NULL) = 0
poll([{fd=4, events=POLLOUT}], 1, 0)    = 1 ([{fd=4, revents=POLLOUT}])
sendto(4, "\3\326\1\0\0\1\0\0\0\0\0\0\tvkontakte\2ru\0\0\1\0\1", 30, MSG_NOSIGNAL, NULL, 0) = 30
poll([{fd=4, events=POLLIN}], 1, 5000)  = 1 ([{fd=4, revents=POLLIN}])
ioctl(4, FIONREAD, [62])                = 0
recvfrom(4, "\3\326\201\200\0\1\0\2\0\0\0\0\tvkontakte\2ru\0\0\1\0\1\300\f"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("172.16.0.23")}, [16]) = 62
close(4)                                = 0

My previous example was based on dig app, which takes slightly different route in terms of system calls.
So it looks like in majority of cases it would be this rule

auditctl -a exit,always -F arch=b64 -S connect -k connect

Followed by ausearch

ausearch -k connect -i|grep saddr|grep "serv:53"|awk '{print $6}'|sort|uniq -c

Answer 2

I wrestled with the very same problem a few days ago, and came up with a very simple method. It is based on the fact that the sending process will be waiting for a DNS response to come, on the same port it sent the request from:

1. Find out the source port of the outgoing DNS request, with iptables -j LOG
2. Use lsof -i UDP:<source_port> to find out which process is waiting for response on that port.

Of course, as the response arrives within milliseconds, you can't do that manually; moreover, even when automated, there's no guarantee that you will be able to query the system before the DNS response arrives, and the sending process dies. That is why before even executing the above steps, i also configure the kernel Traffic Controller to delay outgoing packets directed to a specific ip/port (using the tc module netem). This allows me to control the time window i have to query the system about which PID is waiting for the DNS response, on the source UDP port obtained in step 1.

I have automated the above steps, including the tc delay, in a small script called ptrap (which is a more general solution, not limited to DNS requests, thus eligible for detection of processes using any TCP/UDP based protocol). With its aid i found out that, in my case, the service contacting the old DNS server was sendmail.

Answer 3

There is atop. There is a kernel module (netatop) and daemon which will make atop track network usage by process.

You should first install atop

Here is how you install the kernel module. This is valid when the post was written but it can become outdated:

sudo apt install linux-headers-$(uname -r) make zlib1g-dev
wget https://www.atoptool.nl/download/netatop-2.0.tar.gz
tar xvf netatop-2.0.tar.gz
cd netatop-2.0
make
sudo make install
sudo modprobe -v netatop

If you have systemd, create the service file netatopd.service file in /etc/systemd/system/. It would contain:

[Unit]
Description=NetAtop Daemon

[Service]
Type=forking
ExecStart=/usr/sbin/netatopd

[Install]
WantedBy=multi-user.target

Now you can enable the daemon:

sudo systemctl enable netatopd

To see live per-process network usage:

sudo atop -n

To see top 3 network-intensive throughout the day:

atopsar -N

man atopsar for more options.

Answer 4

There are many options to netstat that show combinations of listening/open sockets over tcp/udp/both. Something like:

$> sudo netstat -pan
Active Internet connections (servers and established)
Proto  Recv-Q Send-Q Local Addr            Foreign Addr           State       PID/Program name
...
tcp    0      1      192.168.66.1:39219    192.168.66.139:2003    SYN_SENT    2045/logstash-forwa

...would have given you a lot of output, but included the source, destination, port numbers, and PID of the process owning those ports.

Answer 5

+1 for Dmitry's answer above; that worked nicely for me:

auditctl -a exit,always -F arch=b64 -F a0=2 -S socket -k SOCKET

To see the resulting entries, I grep the log file for that "-k" string

grep SOCKET /var/log/audit/audit.log

To get just the interesting fields,

grep SOCKET /var/log/audit/audit.log | \
  cut -d' ' -f 4- | \
  sed "s|^|@\n|g;s| |\n|g" | \
  grep -E "^((exe|uid|comm)=|@)" | \
  tr '\n@' ' \n' |\
  sort -u

(explanation: cut -d' ' -f 4- -> chop the line into fields using space (-d' ') as delimiter, show fields fourth to last ( 4- ) )

(explanation: sed "s|^|@\n|g;s| |\n|g" -> edit line, prepend '@' char-plus-newline to start of line, change spaces to newlines)

(explanation: grep -E "^((uid|comm|exe)=|@)" -> as each field of the original line is now on it's own line, pick out the interesting fields: user-id, command, executable - and the line-start '@' char.)

(explanation: tr '\n@' ' \n' -> now having only the wanted fields, turn the newlines back into spaces, and the prepended '@' back into a newline (which rejoins the fields into one line)

(explanation: sort -u -> sort lines, show only unique lines)

gives me:

uid=0 comm="atop" exe="/usr/bin/atop" 
uid=0 comm="http" exe="/usr/lib/apt/methods/http" 
uid=0 comm="links" exe="/usr/bin/links" 
uid=0 comm="ntpdate" exe="/usr/sbin/ntpdate" 
uid=0 comm="ufdbguardd" exe="/usr/local/ufdbguard/bin/ufdbguardd" 
uid=1000 comm=536F636B657420546872656164 exe="/usr/lib/firefox/firefox" 
uid=1000 comm="clock-applet" exe="/usr/lib/mate-panel/clock-applet" 
uid=1000 comm="pool" exe="/usr/lib/mate-panel/clock-applet" 
uid=105 comm="http" exe="/usr/lib/apt/methods/http" 
uid=105 comm="https" exe="/usr/lib/apt/methods/https" 
uid=135 comm="unbound" exe="/usr/sbin/unbound" 
uid=13 comm="squid" exe="/usr/src/squid-4-master/src/squid" 
uid=1 comm="debsecan" exe="/usr/bin/python2.7" 

Commands containing spaces are encoded in simple ascii-to-hex method (see audit_logging.c ). To decode, replace "FF" with "ÿ" and recode that from html to ascii :

grep SOCKET /var/log/audit/audit.log | \
  cut -d' ' -f 4- | sed "s|^|@\n|g;s| |\n|g" | \
  grep -E "^((exe|uid|comm)=|@)" | tr '\n@' ' \n' | \
  sort -u  | sed "s|^[^=]*=||g;s| [^ ]*=| |g" | \
  while read U C E ; do \
    echo "$C" | grep -q '"' || \
      { C=\"`echo $C | sed "s|\(..\)|\&#x\1;|g" | recode h4..u8`\" ; } ; \
    echo "uid=$U comm=$C exe=$E" ; 
  done

(explanation: sed "s|^[^=]=||g;s| [^ ]=| |g" -> edit away the 'xxx=' part of the lines - first: line-start (^) followed by any-char-except-'=' is replaced with blank, then space followed by any-char-except-' ' replaced with space)

(explanation: while read U C E ; do ... done -> loop over each line, reading in each of out three bits of data into U,C,E (userid, command, executable))

(explanation: echo "$C" | grep -q '"' || -> test the command field to see if it contains a doublequote - if not ('||') then do the following: )

(explanation: { C=\"echo $C | sed "s|\(..\)|\&#x\1;|g" | recode h4..ascii\" ; } -> print the command string, edit each pair of chars 'FF' to be 'ÿ', then pass through gnu 'recode' to turn them from html entities into ascii chars.)

(explanation: echo "uid=$U comm=$C exe=$E" -> print out the modified line)

This gives me output (just showing the decoded line):

uid=1000 comm="Socket Thread" exe="/usr/lib/firefox/firefox

/ j

Answer 6

lsof would be an appropriate tool to monitor a specific port and determine the PID that's generating traffic on it. For example here I'm monitoring on a server the DNS/domain TCP port 53, so that I can determine which PID is causing the DNS lookup:

$ lsof -PniTCP:53 -r 1 | grep :53

Now if I were to send some curl traffic to the DNS server:

$ curl -v telnet://192.168.3.182:53
* About to connect() to 192.168.3.182 port 53 (#0)
*   Trying 192.168.3.182...
* Connected to 192.168.3.182 (192.168.3.182) port 53 (#0)

we'd see this type of output from the above command:

curl    4953 root    3u  IPv4 197807920      0t0  TCP 192.168.23.48:50688->192.168.3.182:53 (ESTABLISHED)
curl    4953 root    3u  IPv4 197807920      0t0  TCP 192.168.23.48:50688->192.168.3.182:53 (ESTABLISHED)
curl    4953 root    3u  IPv4 197807920      0t0  TCP 192.168.23.48:50688->192.168.3.182:53 (ESTABLISHED)

How it works

The above command that watches for port 53 traffic works by putting lsof into a repeating loop that runs every 1 sec, -r 1. We then tell lsof to report on only traffic that uses port 53, iTCP:53. The -Pn instructs lsof to display hostnames and ports as numbers, and not actual names.

We then use a grep to read the output coming from lsof and filter the output so that we only see :53 port traffic.

The PID of the process that's sending the traffic is in the output being shown by lsof as well. The 2nd column shows the PID, 4953.