20

Is there a method in Linux to check how many ephemeral ports are left available? I occasionally see "Address already in use" errors as a result of running out of ephemeral ports. A machine reboot will resolve this but it would be better to catch it before it happens.

JMc
  • 301
  • 1
  • 2
  • 3
  • If you are hitting errors like that I would suggest that either you are either abusing the system by not using the right software or architecture for a job, or your software is misbehaving or misconfigured. Perhaps your timeouts are too long for your application or something is leaving connections open without using them? – Caleb Apr 20 '11 at 10:20
  • 2
    There are many valid applications that require additional ephemeral ports beyond OS defaults. – GregB Sep 16 '13 at 22:12

2 Answers2

32

The ephermal port range is specified in /proc/sys/net/ipv4/ip_local_port_range. You can probably extend it to run from 16k to 64k.

You can see the number of open connections using netstat -an. Sockets may be stuck in TIME_WAIT state if you are opening and closing a lot of connections. In some places this is unavoidable, but you may need to consider if you need a pool of connection if this is the case.

If TIME_WAIT is the problem, you can set net.ipv4.tcp_tw_reuse / net.ipv4.tcp_tw_recycle to speed up connection turnover.

Caleb
  • 11,583
  • 4
  • 35
  • 49
Sean
  • 329
  • 2
  • 2
  • 1
    +1, thanks for taking the time to give this guy exact details. – Caleb Apr 20 '11 at 10:45
  • We have a range of 32800 to 61000. We just find that once these have been used the OS will not use them again. This is expected behaviour, but I would expect the OS to start again at the beginning once it reaches the last available port. This doesn't seem to happen though. Also, just to note, this isn't a hugely regular occurance. It is intermittent but we do have a large number of servers. – JMc Apr 20 '11 at 10:45
  • Note that [`net.ipv4.tcp_tw_recycle` was removed in Linux  4.12](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4396e46187ca5070219b81773c4e65088dac50cc) – Anon Aug 11 '18 at 05:11
  • 1
    To comply with [RFC 6335](https://tools.ietf.org/html/rfc6335) `/proc/sys/net/ipv4/ip_local_port_range` needs to be a subset of 49152-65535. So reducing the lower end of the range to anything less than 49152 does come at a certain risk. – kasperd Feb 03 '19 at 21:19
  • never user net.ipv4.tcp_tw_recycle nor net.ipv4.tcp_tw_reuse except if you're desperate and know exactly what you're doing. You're exposing your service to potential extrems issue. – Kiwy Aug 21 '19 at 09:27
3

Bear in mind that this limit applies per unique (source IP, peer IP, peer port) tuple. Therefore you will need to group the output of netstat/ss by each of these tuples, and check how close each group is to the connection limit.

This post explains how you can do this grouping in more detail. To check how close each group is to the limit in Ruby, you can process ss output like:

#!/usr/bin/ruby

first_port, last_port = IO.read('/proc/sys/net/ipv4/ip_local_port_range').split.map(&:to_i)
ephemeral_port_max = last_port - first_port + 1
ephemeral_port_warning = ephemeral_port_max / 3 * 2

conns = `ss --numeric --tcp state connected "( sport >= :#{first_port} and sport <= :#{last_port} )"`

groups = Hash.new(0)
conns.lines.each do |conn|
  state, recvq, sendq, local, peer = conn.split
  local_ip, local_port = local.split(':')
  group = [local_ip, peer]
  groups[group] += 1
end

groups_requiring_warning =
  groups.select { |k, v| v > ephemeral_port_warning }
  .to_a
  .sort_by { |v1, v2| v1[1] <=> v2[1] } # Sort groups in descending order of number of connections

groups_requiring_warning.each do |group, used_port_count|
  puts "Connections from #{group[0]} to #{group[1]} "\
    "have used #{used_port_count} ephemeral ports out of #{ephemeral_port_max} max"\
    "(#{((used_port_count.to_f / ephemeral_port_max) * 100).round(2)}% used)"
end
Will Sewell
  • 131
  • 6