Unable to NAT TFTP traffic because iptables is not forwarding the return connection to the client despite TFTP helper creating an expectation

Question

The Problem

I have a TFTP server (Machine 'S') and a TFTP client (Machine 'C') on different subnets. They are connected via a router( Machine 'R'). All 3 machines are Debian 9/Stretch. The router is running iptables and is set to masquerade connections from the client's network to the server's network. I have configured iptables to use the Netfilter TFTP helper for tftp connections going to the TFTP server.

The trouble I'm having is that the TFTP helper sets up an expectation for the return tftp connection (as expected) but, despite this, only traffic from port 69 on the TFTP server is getting translated and sent back to the client. So only the regular MASQUERADE connection tracking is being used even though the conntrack table shows the expected return connection. According to RFC1350, the server is supposed to choose a random source port for its communication and direct it to the port that the client used as a source port originally (whew...).

The result is the that the router NATs the connection from the client to the server, sets up a translation rule for the return connection and happily waits for a return packet from the server with source port=69 that never arrives.

The Setup

Addresses are made up for clarity:

TFTP Server(S): 1.1.1.1
TFTP Client(C): 2.2.2.1
Router(R): 1.1.1.2 / 2.2.2.2

Iptables on the router has the following rules. All tables have default ACCEPT policy:

======== RAW Table ========
Chain PREROUTING (policy ACCEPT 464K packets, 432M bytes)
 pkts bytes target     prot opt in     out     source       destination
   59  2504 CT         udp  --  *      *       0.0.0.0/0    0.0.0.0/0       udp dpt:69 CT helper tftp

Chain OUTPUT (policy ACCEPT 280K packets, 36M bytes)
 pkts bytes target     prot opt in     out     source       destination

======== NAT Table ========
Chain POSTROUTING (policy ACCEPT 398 packets, 40794 bytes)
 pkts bytes target     prot opt in     out     source       destination
 5678  349K MASQUERADE  all  --  *     enp1s0  0.0.0.0/0    0.0.0.0/0

Once the TFTP client is trying to connect, conntrack -L shows the following:

udp      17 28 src=2.2.2.1 dst=1.1.1.1 sport=45084 dport=69 [UNREPLIED] src=1.1.1.1 dst=1.1.1.2 sport=69 dport=45084 mark=0 helper=tftp use=1

conntrack -L EXPECT:

298 proto=17 src=1.1.1.1 dst=1.1.1.2 sport=0 dport=45084 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=2.2.2.1 master-dst=1.1.1.1 sport=45084 dport=69 class=0 helper=tftp

As you can see, the TFTP helper rule is working properly and is triggered once the client attempts its connection. As you can also see, the expectation created in the EXPECT table has source port 0, which I assume means "any port". But, as you'll see, the connection is only routed back to the client if the source port from the server is port 69 (regular old NAT)! Why is this? This is not the correct behaviour as far as I can tell.

I won't clutter this post anymore if I can avoid it, but what's shown by tcpdump udp and host 1.1.1.1 confirms exactly what iptables and conntrack are showing me.

I did this same setup on several Debian 8/Jessie setups about a year ago and the TFTP helper worked as expected and I never had any issues. Can anyone hlep me figure out if I'm doing something wrong? Is the issue with the TFTP helper? Why would its behaviour have changed from Debian 8/Jessie?

@A.B "that's only the usual udp NAT" sorry I had a lot of trouble articulating what I was seeing. I'll try to clarify it. As for your comments, I assume the automatic helper selection is off by default (nf_conntrack_helper=0) but I'll double check. The only other rule anywhere is a DNAT rule for TCP:80 on the router. All tables use ACCEPT policy. I'll attach some tcpdump output when I get access to that system again. It shows that despite the entry in the EXPECT table, only traffic from 1.1.1.1:69 is actually being translated and forwarded back to the client. Thanks a lot for your suggestions. — succulent_headcrab, Dec 08 '18 at 15:20
Yes, I tried rewording the question to point out that what I'm getting is only the default NAT behaviour despite the TFTP helper's conntrack changes. I can't figure out how to re-word the title though. — succulent_headcrab, Dec 08 '18 at 15:26
Yup, already done. I was obviously too tired and frustrated at this point to write a coherent question. I edited as best I could to try to start in a neutral place instead of starting down the wrong path right off the bat. — succulent_headcrab, Dec 08 '18 at 15:36
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/86793/discussion-between-a-b-and-succulent-headcrab). — A.B, Dec 08 '18 at 15:36

A.B · Answer 1 · 2018-12-08T18:19:28.847

If my guess is right, then TL;DR, on router do this:

modprobe nf_nat_tftp

Either the previous jessie kernel (3.16) auto-loaded nf_nat_tftp, either a script did it, but it appears it's not the case anymore.

Anyway, if that didn't help, here's how to easily reproduce the setup in OP's question, allowing to do any kind of test easily on any Linux system (but remember it's only virtualizing network here, nothing else). I hope this can help if it didn't solve.

Needed: root user, ip netns, atftpd and atftp (or equivalent server and client software).

From scratch there's no reason to have a TFTP helper loaded on the router (so here on the testing host).

lsmod | grep _tftp

should return nothing. Let's suppose it returned nothing or returned only nf_conntrack_tftp from previous rules usage, but not nf_nat_tftp (just rmmod nf_nat_tftp if present, to follow the line of thoughts below).

Using these commands to initialize the namespaces:

ip netns del tftp || :
ip netns del router || :
ip netns del client || :

ip netns add client
ip netns add router
ip netns add tftp
ip -n tftp link add eth0 type veth peer netns router name tftp0
ip -n client link add eth0 type veth peer netns router name client0
ip -n router link set client0 up
ip -n router link set tftp0 up
ip -n tftp link set eth0 up
ip -n client link set eth0 up
ip -n tftp addr add dev eth0 1.1.1.1/24
ip -n router addr add dev tftp0 1.1.1.2/24 
ip -n router addr add dev client0 2.2.2.2/24 
ip -n client addr add dev eth0 2.2.2.1/24
ip -n client route add default via 2.2.2.2
ip netns exec router sh -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'
ip netns exec router sh -c 'echo 0 > /proc/sys/net/netfilter/nf_conntrack_helper' # this one might be global on former kernels and might need to be executed without "ip netns exec router"
ip netns exec router iptables -t nat -A POSTROUTING -o tftp0 -j MASQUERADE
ip netns exec router iptables -t raw -A PREROUTING -p udp --dport 69 -j CT --helper tftp

That last command should automatically trigger the loading of nf_conntrack_tftp:

# uname -r
4.19.7
# lsmod | grep _tftp
nf_conntrack_tftp      16384  1
nf_conntrack          163840  20 xt_conntrack,[...],nf_nat,nf_conntrack_tftp,[...],nf_nat_ipv4,[...]

(Depending on kernel, there might be nf_conntrack_ipv4 instead of nf_conntrack appearing)

Prepare files:

mkdir -p /tmp/tftp
echo test > /tmp/tftp/test.txt
mkdir -p /tmp/client

Term1:

ip netns exec router tcpdump -e -n -s0 -i any ip

Term2:

ip netns exec tftp atftpd --daemon --no-fork /tmp/tftp

Term3:

ip netns exec client atftp -g -l /tmp/client/test.txt -r test.txt 1.1.1.1

Will display:

timeout: retrying...
^Ctftp: unknown error.
tftp: aborting

Term 1 will show something like:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
17:46:52.958631  In 46:b9:b2:13:ec:51 ethertype IPv4 (0x0800), length 61: 2.2.2.1.36231 > 1.1.1.1.69:  17 RRQ "test.txt" octet
17:46:52.958719 Out 1a:0c:61:3d:f9:00 ethertype IPv4 (0x0800), length 61: 1.1.1.2.36231 > 1.1.1.1.69:  17 RRQ "test.txt" octet
17:46:52.959324  In d6:49:0e:de:b7:27 ethertype IPv4 (0x0800), length 53: 1.1.1.1.38635 > 1.1.1.2.36231: UDP, length 9
17:46:52.959397 Out 1a:0c:61:3d:f9:00 ethertype IPv4 (0x0800), length 81: 1.1.1.2 > 1.1.1.1: ICMP 1.1.1.2 udp port 36231 unreachable, length 45
17:46:57.960151  In 46:b9:b2:13:ec:51 ethertype IPv4 (0x0800), length 61: 2.2.2.1.36231 > 1.1.1.1.69:  17 RRQ "test.txt" octet
17:46:57.960213 Out 1a:0c:61:3d:f9:00 ethertype IPv4 (0x0800), length 61: 1.1.1.2.36231 > 1.1.1.1.69:  17 RRQ "test.txt" octet
17:46:57.960834  In d6:49:0e:de:b7:27 ethertype IPv4 (0x0800), length 53: 1.1.1.1.44001 > 1.1.1.2.36231: UDP, length 9
17:46:57.960932 Out 1a:0c:61:3d:f9:00 ethertype IPv4 (0x0800), length 81: 1.1.1.2 > 1.1.1.1: ICMP 1.1.1.2 udp port 36231 unreachable, length 45

Now after doing (on the router, so here on the testing host) modprobe nf_nat_tftp, the client command in Term3 will succeed and Term1 will show:

17:54:11.142487  In 46:b9:b2:13:ec:51 ethertype IPv4 (0x0800), length 61: 2.2.2.1.49514 > 1.1.1.1.69:  17 RRQ "test.txt" octet
17:54:11.142556 Out 1a:0c:61:3d:f9:00 ethertype IPv4 (0x0800), length 61: 1.1.1.2.49514 > 1.1.1.1.69:  17 RRQ "test.txt" octet
17:54:11.143090  In d6:49:0e:de:b7:27 ethertype IPv4 (0x0800), length 53: 1.1.1.1.36339 > 1.1.1.2.49514: UDP, length 9
17:54:11.143146 Out 96:c2:ee:fb:cc:07 ethertype IPv4 (0x0800), length 53: 1.1.1.1.36339 > 2.2.2.1.49514: UDP, length 9
17:54:11.143226  In 46:b9:b2:13:ec:51 ethertype IPv4 (0x0800), length 48: 2.2.2.1.49514 > 1.1.1.1.36339: UDP, length 4
17:54:11.143262 Out 1a:0c:61:3d:f9:00 ethertype IPv4 (0x0800), length 48: 1.1.1.2.49514 > 1.1.1.1.36339: UDP, length 4

Looking at conntrack expectations and flows (examples not matching above examples):

without nf_nat_tftp:

# ip netns exec router sh -c 'conntrack -E & conntrack -E expect'
    [NEW] 300 proto=17 src=1.1.1.1 dst=1.1.1.2 sport=0 dport=56876 mask-src=0.0.0.0 mask-dst=0.0.0.0 sport=0 dport=65535 master-src=2.2.2.1 master-dst=1.1.1.1 sport=56876 dport=69 class=0 helper=tftp
    [NEW] udp      17 30 src=2.2.2.1 dst=1.1.1.1 sport=56876 dport=69 [UNREPLIED] src=1.1.1.1 dst=1.1.1.2 sport=69 dport=56876 helper=tftp
[DESTROY] 299 proto=17 src=1.1.1.1 dst=1.1.1.2 sport=0 dport=56876 mask-src=0.0.0.0 mask-dst=0.0.0.0 sport=0 dport=65535 master-src=2.2.2.1 master-dst=1.1.1.1 sport=56876 dport=69 class=0 helper=tftp
    [NEW] udp      17 30 src=1.1.1.1 dst=1.1.1.2 sport=58241 dport=56876 [UNREPLIED] src=1.1.1.2 dst=1.1.1.1 sport=56876 dport=58241
^Cconntrack v1.4.4 (conntrack-tools): conntrack v1.4.4 (conntrack-tools): 2 flow events have been shown.
2 expectation events have been shown.

Above, the 2nd flow (last line) doesn't refer to 2.2.2.1.

With nf_nat_tftp:

# ip netns exec router sh -c 'conntrack -E & conntrack -E expect'
    [NEW] 300 proto=17 src=1.1.1.1 dst=1.1.1.2 sport=0 dport=38115 mask-src=0.0.0.0 mask-dst=0.0.0.0 sport=0 dport=65535 master-src=2.2.2.1 master-dst=1.1.1.1 sport=38115 dport=69 class=0 helper=tftp
    [NEW] udp      17 30 src=2.2.2.1 dst=1.1.1.1 sport=38115 dport=69 [UNREPLIED] src=1.1.1.1 dst=1.1.1.2 sport=69 dport=38115 helper=tftp
[DESTROY] 299 proto=17 src=1.1.1.1 dst=1.1.1.2 sport=0 dport=38115 mask-src=0.0.0.0 mask-dst=0.0.0.0 sport=0 dport=65535 master-src=2.2.2.1 master-dst=1.1.1.1 sport=38115 dport=69 class=0 helper=tftp
    [NEW] udp      17 30 src=1.1.1.1 dst=1.1.1.2 sport=35725 dport=38115 [UNREPLIED] src=2.2.2.1 dst=1.1.1.1 sport=38115 dport=35725
 [UPDATE] udp      17 30 src=1.1.1.1 dst=1.1.1.2 sport=35725 dport=38115 src=2.2.2.1 dst=1.1.1.1 sport=38115 dport=35725
^Cconntrack v1.4.4 (conntrack-tools): 2 expectation events have been shown.
conntrack v1.4.4 (conntrack-tools): 3 flow events have been shown.

The 2nd flow does refer to 2.2.2.1.

So it appears that while nf_conntrack_tftp is good enough to allow the 2nd flow usage for example with -m ctstate --ctstate RELATED in a firewall, nf_nat_tftp is still required to also actually alter in the 2nd flow the destination IP (and perhaps sometimes port) when NAT is in use. AFAIK, at least in recent kernels, nothing will trigger the loading of the companion NAT TFTP helper in addition to the conntrack TFTP helper: it has to be loaded manually.

Unable to NAT TFTP traffic because iptables is not forwarding the return connection to the client despite TFTP helper creating an expectation

The Problem

The Setup

1 Answers1

Linked