How can a server accept incoming traffic to port 443 if it isn't listed in netstat?

Question

I have a server that is currently accepting HTTPS requests and the only thing that shows with netstat -plnt is this:

elijahlynn@web:~$ sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 10.218.108.1:53         0.0.0.0:*               LISTEN      10242/dnsmasq   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1111/sshd       
tcp6       0      0 fd42:9324:ab98:50fb::53 :::*                    LISTEN      10242/dnsmasq   
tcp6       0      0 fe80::c024:c5ff:fe68:53 :::*                    LISTEN      10242/dnsmasq   
tcp6       0      0 :::22                   :::*                    LISTEN      1111/sshd

And sudo lsof -i :443 returns nothing as well.

I have tried variations of the ss tool as well but no luck in seeing anything listening.

This server uses iptables to forward traffic to an LXC "haproxy" container but from my understanding, first the OS needs to bind a port to a dameon or application to begin with. The fact that this isn't showing any ports in the LISTEN state yet still serves traffic is really challenging my understanding of how things work.

Is there any way that this is possible for a program to be listening on a port but not showing with netstat, lsof or ss?

UPDATE:

elijahlynn@web:~$ sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             10.0.0.0/8           state NEW,RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps /* generated for LXD network lxdbr0 */

UPDATE2:

elijahlynn@web:~$ sudo ss -lptn | column -t
State   Recv-Q  Send-Q  Local                                Address:Port  Peer                                 Address:Port
LISTEN  0       32      10.218.108.1:53                      *:*           users:(("dnsmasq",pid=10242,fd=9))
LISTEN  0       128     *:22                                 *:*           users:(("sshd",pid=1111,fd=3))
LISTEN  0       32      fd42:9324:ab98:50fb::1:53            :::*          users:(("dnsmasq",pid=10242,fd=13))
LISTEN  0       32      fe80::c024:c5ff:fe68:999e%lxdbr0:53  :::*          users:(("dnsmasq",pid=10242,fd=11))
LISTEN  0       128     :::22                                :::*          users:(("sshd",pid=1111,fd=4))

UPDATE3:

elijahlynn@web:~$ sudo lsof -i -n  | column -t
COMMAND  PID    USER        FD   TYPE  DEVICE     SIZE/OFF  NODE  NAME
sshd     1111   root        3u   IPv4  786        0t0       TCP   *:ssh                                 (LISTEN)
sshd     1111   root        4u   IPv6  788        0t0       TCP   *:ssh                                 (LISTEN)
dnsmasq  10242  lxd         4u   IPv4  343469983  0t0       UDP   *:bootps
dnsmasq  10242  lxd         6u   IPv6  343469985  0t0       UDP   *:dhcpv6-server
dnsmasq  10242  lxd         8u   IPv4  343469988  0t0       UDP   10.218.108.1:domain
dnsmasq  10242  lxd         9u   IPv4  343469989  0t0       TCP   10.218.108.1:domain                   (LISTEN)
dnsmasq  10242  lxd         10u  IPv6  343469990  0t0       UDP   [fe80::c024:c5ff:fe68:999e]:domain
dnsmasq  10242  lxd         11u  IPv6  343469991  0t0       TCP   [fe80::c024:c5ff:fe68:999e]:domain    (LISTEN)
dnsmasq  10242  lxd         12u  IPv6  343469992  0t0       UDP   [fd42:9324:ab98:50fb::1]:domain
dnsmasq  10242  lxd         13u  IPv6  343469993  0t0       TCP   [fd42:9324:ab98:50fb::1]:domain       (LISTEN)

Just added some more output from `ss` and `lsof`. – Elijah Lynn May 07 '20 at 17:39 — Elijah Lynn, May 07 '20 at 17:39

score 2 · Answer 1 · answered May 07 '20 at 03:47

iptables can do a lot of voodoo to traffic -- look for a redirect rule. And LXC can further muddy the waters with different network namespaces -- ip netns list

[root@r510-main ~]# ip -all netns exec netstat -plnt

netns: qrouter-a0291178-a7d9-4419-a47e-47d8bdfe6fa2
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:9697            0.0.0.0:*               LISTEN      11202/haproxy

netns: qdhcp-4addb334-d787-4d0c-a1bf-d3696c67cb15
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      11190/dnsmasq
tcp6       0      0 ::1:53                  :::*                    LISTEN      11190/dnsmasq
tcp6       0      0 fd00:...:1200:f81:53    :::*                    LISTEN      11190/dnsmasq
tcp6       0      0 fe80::...fe9a:53        :::*                    LISTEN      11190/dnsmasq

netns: qdhcp-0c964c4d-3cd3-415b-9f42-61206fff0328
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      11186/dnsmasq
tcp        0      0 172.24.18.2:53          0.0.0.0:*               LISTEN      11186/dnsmasq
tcp6       0      0 ::1:53                  :::*                    LISTEN      11186/dnsmasq
tcp6       0      0 fd00:...:12ff:f81:53    :::*                    LISTEN      11186/dnsmasq
tcp6       0      0 fe80::...fe88:53        :::*                    LISTEN      11186/dnsmasq

Yet, from dom-0 lsof -n -P -i :53 returns nothing.

[root@r510-main ~]# lsof -n -P -i :53    
[root@r510-main ~]# ip -all netns exec lsof -n -P -i :53

netns: qrouter-a0291178-a7d9-4419-a47e-47d8bdfe6fa2

netns: qdhcp-4addb334-d787-4d0c-a1bf-d3696c67cb15
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dnsmasq 11190 nobody    4u  IPv4 105932      0t0  UDP 127.0.0.1:53 
dnsmasq 11190 nobody    5u  IPv4 105933      0t0  TCP 127.0.0.1:53 (LISTEN)
dnsmasq 11190 nobody    6u  IPv6 105934      0t0  UDP [fe80::...:fe9a:9bd2]:53 
dnsmasq 11190 nobody    7u  IPv6 105935      0t0  TCP [fe80::...:fe9a:9bd2]:53 (LISTEN)
dnsmasq 11190 nobody    8u  IPv6 105936      0t0  UDP [fd00:...:9bd2]:53 
dnsmasq 11190 nobody    9u  IPv6 105937      0t0  TCP [fd00:...:9bd2]:53 (LISTEN)
dnsmasq 11190 nobody   10u  IPv6 105938      0t0  UDP [::1]:53 
dnsmasq 11190 nobody   11u  IPv6 105939      0t0  TCP [::1]:53 (LISTEN)

netns: qdhcp-0c964c4d-3cd3-415b-9f42-61206fff0328
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dnsmasq 11186 nobody    6u  IPv4  94770      0t0  UDP 172.24.18.2:53 
dnsmasq 11186 nobody    7u  IPv4  94771      0t0  TCP 172.24.18.2:53 (LISTEN)
dnsmasq 11186 nobody    8u  IPv4  94772      0t0  UDP 127.0.0.1:53 
dnsmasq 11186 nobody    9u  IPv4  94773      0t0  TCP 127.0.0.1:53 (LISTEN)
dnsmasq 11186 nobody   10u  IPv6  94774      0t0  UDP [fe80::...:fe88:a0b5]:53 
dnsmasq 11186 nobody   11u  IPv6  94775      0t0  TCP [fe80::...:fe88:a0b5]:53 (LISTEN)
dnsmasq 11186 nobody   12u  IPv6  94776      0t0  UDP [fd00:...:fe88:a0b5]:53 
dnsmasq 11186 nobody   13u  IPv6  94777      0t0  TCP [fd00:...:fe88:a0b5]:53 (LISTEN)
dnsmasq 11186 nobody   14u  IPv6  94778      0t0  UDP [::1]:53 
dnsmasq 11186 nobody   15u  IPv6  94779      0t0  TCP [::1]:53 (LISTEN)

Thanks, testing your commands now and reading `man ip` as both commands don't return anything right now, both `ip -all netns exec netstat -plnt` and `ip -all netns exec lsof -n -P -i :443`. The later returns with 53 and dnsmasq though. Reading more about network namespaces ... — Elijah Lynn, May 07 '20 at 16:39
So it sounds like `iptables` gets first dibs on things and if it forwards to another IP then it will never show up in any of the IP tools (ip, lsof, netstat). Is that right? — Elijah Lynn, May 07 '20 at 16:45

score 0 · Answer 2 · answered May 07 '20 at 03:33

0

Maybe you want to use nmap for scanning port on your server, here an example :

$ nmap -p443 ip-address

Starting Nmap 7.80 ( https://nmap.org ) at 2020-05-07 04:33 a/B
Nmap scan report for some IP
Host is up (0.011s latency).

PORT    STATE SERVICE
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 2.45 seconds

answered May 07 '20 at 03:33

YonzLeon

168
5

Thanks, I do see 443 come up with nmap, and in a closed state. Doesn't there always have to be a LISTEN state on the port in order to bind it with the application that will handle it? – Elijah Lynn May 07 '20 at 16:37

score 0 · Answer 3 · edited May 28 '22 at 00:27

0

Check if you have anything in the nat table: sudo iptables -t nat -L

The filter table is the only thing shown in your original output, while if iptables is redirecting 443 to some listening port present, it will be in the NAT table.

edited May 28 '22 at 00:27

Chris Buechler

2,938
14
18

answered Apr 19 '22 at 16:46

DionisisL

1

How can a server accept incoming traffic to port 443 if it isn't listed in netstat?

3 Answers3