So port is not unique by machine at all?

Question

$ lsof

memcached 15844 root   28u  IPv6     113604       TCP *:11051 (LISTEN)
memcached 15844 root   29u  IPv4     113605       TCP *:11051 (LISTEN)
memcached 15844 root   30u  IPv6     113609       UDP *:11051 
memcached 15844 root   31u  IPv4     113610       UDP *:11051 

Is it by IP, by protocol or by any other dimensions?

Answer 1

The same port number can be used once for TCP and once for UDP, and each of these exists on IPv4 and IPv6.

TCP and UDP port numbers are an entirely different number spaces, they just happen to be assigned in pairs and applications that need both usually use the same number on each.

IPv4 and IPv6 are somewhat distinct protocols; in principle, an IPv6 application can accept IPv4 connections, but it is generally considered good style to have two separate sockets.

Answer 2

Each entry in the netstat or lsof -i output is called a socket.

Each socket is made unique by a combination of:

• protocol family (e.g. IPv4 or IPv6)
• protocol (e.g. TCP or UDP)
• local address
• local port
• remote address
• remote port

This combination of variables is called the connection tuple.

Consider ssh.

If your machine is running sshd, you will see an entry in the netstat output for the server that is running on port 22, listening and waiting for new connections.

# netstat -tnl | grep 22
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN

But you will also see entries for each active connection.

# netstat -tn
tcp        0      0 192.168.x.y:22          192.168.x.a:ppppp       ESTABLISHED
tcp        0      0 192.168.x.y:22          192.168.x.b:qqqqq       ESTABLISHED

So the above output shows port 22 is being used 3 times on the same machine.

In this case, each connection is unique because each one has a different remote address or port. Note that even if only one of those is different, it still makes the socket unique.

But it can also be unique by local address or port.

For example, you could be using Apache IP-based virtual hosts to show a different site depending on the host name or IP address:

# netstat -tnlp | grep 80
tcp        0      0 1.2.3.4:80              0.0.0.0:*               LISTEN      9876/apache2
tcp        0      0 1.2.3.5:80              0.0.0.0:*               LISTEN      9876/apache2

Here, there are two sockets listening on port 80.

In this example, if traffic comes in with a destination address of 1.2.3.4, it will be delivered to the first socket; if the destination is 1.2.3.5 it goes to the second one.

The server has to do this explicitly however. The default for a server is to set the local address to 0.0.0.0, also written *, which means there will only be one listening socket on that port, and it will accept all incoming traffic on that port.

As you have discovered, it is also unique per protocol.

For example, the BIND DNS server uses both TCP and UDP:

# netstat -nlp | grep named
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      567/named
tcp        0      0 192.168.x.y:53          0.0.0.0:*               LISTEN      567/named
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN      567/named
udp        0      0 127.0.0.1:53            0.0.0.0:*                           567/named
udp        0      0 192.168.x.y:53          0.0.0.0:*                           567/named

Each connection is also unique per protocol family, meaning IPv4 and IPv6 can be distinct.

This depends on whether the programmer asked for an IPv4-only socket, an IPv6-only socket, or a combined IPv4/IPv6 socket.

On Linux an IPv6 socket also accepts IPv4 traffic unless the programmer tells it not to, and therefore only appears once in the netstat output.

$ port=55555
$ nc -6 -l $port || echo "Error listening on port $port" &
$ netstat -tnl | grep $port
tcp6       0      0 :::55555                :::*                    LISTEN     
$ nc -z 127.0.0.1 $port && echo "Port $port is accepting IPv4 connections"
Port 55555 is accepting IPv4 connections

But the memcached source code shows it is explicitly asking for an IPv6-only socket, which is why IPv4 and IPv6 appear separately in your example.

error = setsockopt(sfd, IPPROTO_IPV6, IPV6_V6ONLY, (char *) &flags, sizeof(flags));