Find out what Linux software is trying to phone home

Question

I have seen some suspicious outbound communications blocked on my VPS, which hosts a small experimental Web site, and want to find out which application is making the attempt. The server is running CentOS and I'm using iptables for the firewall. Here is an excerpt from my iptables rule set:

# create a new chain for rejecting certain ip addresses  
-N input_tcp_deny  
-A input_tcp_deny -m limit --limit 5/min -j LOG --log-prefix "tcp/ip input denied " --log-level 6  
-A input_tcp_deny -j REJECT  

# create a new chain for rejecting certain ip addresses  
-N output_tcp_deny  
-A output_tcp_deny -m limit --limit 5/min -j LOG --log-prefix "tcp/ip output denied " --log-level 4  
-A output_tcp_deny -j REJECT  

-A INPUT -p tcp -s 171/8 -j input_tcp_deny  
-A OUTPUT -p tcp -d 171/8 -j output_tcp_deny  
-A INPUT -p tcp -s 141/8 -j input_tcp_deny  
-A OUTPUT -p tcp -s 141/8 -j output_tcp_deny  
-A INPUT -p tcp -s 103/8 -j input_tcp_deny  
-A OUTPUT -p tcp -d 103/8 -j output_tcp_deny

iptbles -nvL output shows:

Chain output_tcp_deny (91 references)  
 pkts bytes target     prot opt in     out     source               destination  
    6   120 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 5/min burst 5 LOG flags 0 level 4 prefix `tcp/ip output denied '  
    6   120 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable

Looking in the iptables log file shows:

Apr  9 00:19:25 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=171.64.65.117 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15407 DF PROTO=TCP SPT=46898 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr  9 00:19:25 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54582 WINDOW=14480 RES=0x00   ACK SYN URGP=0  
Apr  9 00:19:25 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.219.155.233 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=80 DPT=54760 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=171.64.65.117 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15408 DF PROTO=TCP SPT=46898 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54582 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.219.155.233 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=80 DPT=54760 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54582 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:55 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:55 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.216.10.182 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=80 DPT=53542 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:56 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:56 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:58 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=103.25.63.44 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43399 DF PROTO=TCP SPT=45638 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr 12 11:48:58 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:59 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=103.25.63.44 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43400 DF PROTO=TCP SPT=45638 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr 12 11:49:01 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=163.178.174.25 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59335 DF PROTO=TCP SPT=37733 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr 12 11:49:02 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=163.178.174.25 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59336 DF PROTO=TCP SPT=37733 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0

Looking up the IP addresses for blocked outbound communications:
- 171.64.65.117 stanford.edu
- 103.25.63.44 Hong Kong Wholesale Data Center LLC

Inbound blocked communications happened very close in time so I looked those up too:
- 141.214.186.162 University Of Michigan
- 141.219.155.233 mtu.edu (University Of Michigan)
- 141.216.10.182 umflint.edu (University Of Michigan)

For anyone who is curious, I also have IPV6 completely blocked (but no logs) and have seen this from ip6tables -nvL:

Chain OUTPUT (policy DROP 8 packets, 536 bytes)

So I have dates, times and ip addresses. But now I would like to find out what open source software is doing this.

I think the answer involves `-j ULOG` and configuring [ulogd](http://rlworkman.net/howtos/ulogd.html). — Gilles 'SO- stop being evil', Apr 22 '14 at 11:06

score 2 · Answer 1 · edited Apr 13 '17 at 12:13

If you have auditd installed, you can use it to log connections:

auditctl -a exit,always -F arch=b64 -S connect -k NETWORK

If you have a 32-bit operation system, use b32 rather than b64.

Now you should be able to look in your auditd logs files in /var/log/ to find the processes establishing outgoing connections.

Unfortunately, the audit.log is a bit cryptic. Here is an example from my logs after I run "nc 8.8.8.8 53":

type=SYSCALL msg=audit(1401890039.928:1080289): arch=c000003e syscall=42 success=no exit=-115 a0=3 a1=df3310 a2=10 a3=7fffd8db6910 items=0 ppid=7310 pid=7725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=pts5 comm="nc" exe="/bin/nc.openbsd" key="NETWORK"
type=SOCKADDR msg=audit(1401890039.928:1080289): saddr=02000035080808080000000000000000

In the type=SYSCALL line, you can see the process id, the cmd "/bin/nc.openbsd" and information about the uid running the process. In the type=SOCKADDR line, you have the saddr field containing information about the ip/tcp connection:

02000035080808080000000000000000

02 = IP
35 = Port 53
08080808 = 8.8.8.8

If you want to search for the IP address 163.178.174.25, you could do the following in python to get the hex representation of this ip.

$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import struct, socket
>>> "%x" % struct.unpack("<L", socket.inet_aton("163.178.174.25"))[0]
'19aeb2a3'

Then, you can grep the audit.log for occurrences of this hex string to find the offending application.

And finally, when you have finished the debugging, remember to turn off the auditd rule, as it can produce a fair amount of logs.

auditctl -d exit,always -F arch=b64 -S connect -k NETWORK

Happy hacking!

Credit to answers on serverfault:

https://serverfault.com/questions/352259/finding-short-lived-tcp-connections-owner-process

https://serverfault.com/questions/192893/how-i-can-identify-which-process-is-making-udp-traffic-on-linux

score 0 · Answer 2 · answered Apr 22 '14 at 16:13

You could use a loop to continuously call lsof to see all of the open network connections, log it to a file then grep through it for port 80. That would show you the process opening the socket and the user it's running as.

It could go something like:

#!/bin/bash
# monitor.sh
while true; do
    date
    lsof -i -P
    sleep 1
done

To use it:

$ chmod +x monitor.sh
$ sudo ./monitor.sh > connections.log

To check it:

$ grep ":80 (" connections.log

By including " (" in grep's pattern to match lsof's output, grep should only match on outbound connections to port 80.

I would grep the log every couple of hours and I would also test it first to make sure the log file wouldn't get too big and hog all my disk space.

`watch` is much nicer than `while true; do...done` – Marcin Apr 22 '14 at 19:30 — Marcin, Apr 22 '14 at 19:30

score 0 · Answer 3 · answered Jun 04 '14 at 14:34

As well as the other good answers (auditd in particular), your situation might call for the following:

Remove the --ulimit parameter from iptables, to make sure you're not missing any connection
Redirect the logs to a different logfile or even better to a different log facility
Configure syslog to catch that facility and redirect to a pipe (see this link for an example)
Write a shell script that reads the content of the pipe and upon receiving an event:
1. runs lsof or netstat -nalp to capture PIDs and offending binaries
2. launches a tcpdump session to capture the traffic
3. e-mails you so you can act quickly :)

My bet is on some harmless piece of software such as a popularity contest, a NTP daemon or a software update. Still I'm curious so do keep us updated please!

score -1 · Answer 4 · answered Apr 16 '14 at 11:28

Having assessed the risk of turning my iptables off for a few minutes, these would be my first steps:

stop the iptables process to allow the open-source app to initiate a connection home (again, only if you feel safe doing this)
run netstat -lntp | less. This list should tell you the destination IPs that your box is connecting to. If you find the ones you have listed above, try and look for the Process ID (PID) for that connection.
run ps aux | less. Try and locate the same PID you have found on step 2 above. You might have to do some digging and play around with the options of this command. This page might be useful.
restart iptables
reboot the server (might not be necessary, but sometimes that is what it takes for the open-source to call home again, unless you already know that the call-home is constant)
check the iptables logs again. Next steps would depend on your findings here.

Because you have only asked on how to identify, rather than block this traffic, I shall stop here; hopefully this will help you a bit.

This will only work if you can catch the connection while it's up, i.e. only if the connection lasts sufficiently long. There's no such thing as “stop the `iptables` process”: `iptables` is the program that changes the settings, not a program that filters traffic; you mean switching the firewall to not block these connections, which doesn't require switching it to being wide open. — Gilles 'SO- stop being evil', Apr 22 '14 at 11:08

Find out what Linux software is trying to phone home

4 Answers4

Linked