netstat shows a listening port with no pid but lsof does not

Question

This question is similar to Network port open, but no process attached?

I've tried everything from there, reviewed the logs, etc... and can't find anything.

My netstat shows a TCP listening port and a UDP port without a pid. When I search lsof for those ports nothing comes up.

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:44231           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:55234           0.0.0.0:*                           -

The following commands display nothing:

lsof | grep 44231
lsof | greo 55234
fuser -n tcp 44231
fuser -n udp 55234

After rebooting, those "same" two connections are there except with new port numbers:

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:45082           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:37398           0.0.0.0:*                           -

And once again, the lsof and fuser commands show nothing.

Any ideas what they are? Should I be concerned about them?

A given solution for the specific problem - https://serverfault.com/questions/1078483/how-to-find-out-what-service-is-listening-on-a-specific-port-of-a-ubuntu-server — CrazyTux, Oct 04 '21 at 20:25

score 23 · Answer 1 · edited Jul 02 '21 at 14:31

23

Some processes/pids are only available to root. Try

sudo netstat -antlp

it should return the pid of every open port that's not in a TIME_WAIT state

or, if you want to know process ID related to specific port (let us say 8765 for example) use the code

netstat -tulpn | grep :8765

edited Jul 02 '21 at 14:31

Bharat Sharma

3
3

answered May 15 '12 at 14:23

user1389651

331
2
2

4

**every open TCP ports** only with this command. UDP ports will not be shown. – petrus Mar 31 '13 at 21:51

score 17 · Answer 2 · answered May 03 '17 at 04:05

Based on hint from @user202173 and others I have been able to use the following to track down the process that owns a port even when it is listed as - in netstat.

Here was my starting situation. sudo netstat shows port with PID/Program of -. lsof -i shows nothing.

$ sudo netstat -ltpna | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  -
$ sudo lsof -i :8785
$

Now let's go fishing. First let's get the inode by adding -e to our netstat call.

$ sudo netstat -ltpnae | awk 'NR==2 || /:8785/'
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp6       0      0 :::8785                 :::*                    LISTEN      199179     212698803   -
tcp6       1      0 ::1:8785                ::1:45518               CLOSE_WAIT  0          0           -

Next use lsof to get the process attached to that inode.

$ sudo lsof | awk 'NR==1 || /212698803/'
COMMAND      PID    TID                USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
envelope_ 145661 145766               drees   15u     IPv6          212698803        0t0        TCP *:8785 (LISTEN)

Now we know the process id so we can look at the process. And unfortunately it's a defunct process. And its PPID is 1 so we can't kill its parent either (see How can I kill a process whose parent is init?). In theory init might eventually clean it up, but I got tired of waiting and rebooted.

$ ps -lf -p 145661
F S UID         PID   PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 Z drees    145661      1  2  80   0 -     0 exit   May01 ?        00:40:10 [envelope] <defunct>

Have been searching for this solution for months now. thank you for this jem — manish Prasad, Mar 22 '19 at 06:23
I've been looking for a solution like this for years. Thank you! One of the downsides here is that if an NFS client or server is bogged down, then `lsof | awk 'NR==1 || /212698803/'` (even with `lsof -N` to show NFS only) can be very slow to respond and may time out. Another downside is that the inode may change while you are troubleshooting this. — Stefan Lasiewski, Nov 15 '19 at 00:34

score 16 · Accepted Answer · answered Sep 14 '11 at 01:04

From data you provided I'd say it's related to some NFS mounts or something using RPC.

you can check with rpcinfo -p for ports that might be used by some of RPC related services.

Here is how it looks on my system

# netstat -nlp | awk '{if ($NF == "-")print $0}'
tcp        0      0 0.0.0.0:55349           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:18049           0.0.0.0:*                           - 

# rpcinfo -p
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  10249  status
    100024    1   tcp  10249  status
    100021    1   udp  18049  nlockmgr
    100021    3   udp  18049  nlockmgr
    100021    4   udp  18049  nlockmgr
    100021    1   tcp  55349  nlockmgr
    100021    3   tcp  55349  nlockmgr
    100021    4   tcp  55349  nlockmgr

If you have this problem and want to force nlockmgr to use specific ports, try this solution: http://www.fclose.com/39625/fixing-ports-used-by-nfs-server. — Ryan Walls, Dec 19 '14 at 08:01

andyortlieb · Answer 4 · 2011-09-13T19:29:50.753

5

I don't know what these are specifically, but kernel modules (NFS for example) do not have a PID to associate with these sockets. Look for something suspect in lsmod.

edited Sep 13 '11 at 19:29

answered Sep 13 '11 at 18:59

andyortlieb

1,052
1
12
25

lsmod returns nothing. This server is an NFS client. That is currently my #1 suspect. – mhost Sep 13 '11 at 19:13
That would explain why the client ports have changed after a new instance of the kernel. – andyortlieb Sep 13 '11 at 19:30
You shouldn't have been downvoted as this is a totally legit answer. It helped me find a case where the other answers (using rpcbind or lsof) didn't help. (And yes, it was NFS.) Thanks! – Peter Hansen Sep 06 '14 at 15:15
Hmm, I wonder why it doesn't assign a PID to the NFS client just so you can see what's up with that ... I guess that'd require it to have a worker thread or something? – SamB Jan 11 '15 at 03:20

score 3 · Answer 5 · answered Dec 11 '13 at 10:56

I dont know if this can be useful. I had the same problem and what I did is the following: First, I called netstat with options -a(all) and -e(extended). With the latter option I can see the Inode associated to the used port. Then, I called lsof |grep with the inode number obtained and I got the PID of process associated to that inode. That worked in my case.

Otheus · Answer 6 · 2021-12-03T21:30:12.423

I've turned @studgeek's excellent answer into a one-line / shell function. Broken into several lines for clarity(ish):

lookup_port_pid() {  
   netstat -lptnae |
   awk -v port="$1" '/^(udp|tcp)/ && (lasti=split($4,hostport,":")) && hostport[lasti] == port { print $8; exit; }' |  { 
     read inode; [[ "$inode" ]] && 
     lsof | 
     awk -v inode=" $inode " 'NR==1 { print; z=index($0,"DEVICE")-10; }  index(substr($0,z),inode) {  print; } '; 
  }; 
}

Usage: lookup_port_pid 443

Maybe it doesn't look like a one-liner anymore, but that's how I wrote it :)

The complex 2nd awk is there to prevent false positives -- ie, inode numbers appearing elsewhere in the lsof output, and is somewhat resistant against changes in the lsof output format, and variations in the line spacing. Normally, something like $6 would do fine, but lsof can sometimes push two columns together without whitespace or a running command could contain whitespace, etc.

score 1 · Answer 7 · answered Sep 13 '11 at 18:29

1

Is there any traffic coming or going from this port, check that with tcpdump -vv -x s 1500 port 37398 -w trace.out Saves your capture in the file trace.out you can then open it with wireshark, or tcpdump -vv port 37398 and see whats going on directly.

Try to telnet to that port use netcat for the udp socket maybe you get some kind of banner that helps.

Get rkhunter and check your system for a backdoor.

Compare the md5 hash of lsof/netstat with the one from your install media, assuming the files where not updatet.

answered Sep 13 '11 at 18:29

Izac

1,758
1
11
11

I did try to locally netcat to both ports and it displays nothing. For the tcp port, it closes if I type anything and then enter. The UDP one only closes if I press Ctrl+C. I have iptables in place, and it does not allow connections to those ports, so unless they are bypassing iptables, I can't imagine something is connecting to them. – mhost Sep 13 '11 at 19:08
what kind of server is it DB,APP.. which software are you using ? – Izac Sep 13 '11 at 19:22
It is a web server running apache and pretty much nothing else other than things like cron and syslog. – mhost Sep 13 '11 at 22:28

netstat shows a listening port with no pid but lsof does not

7 Answers7

Linked

Related