4

I am trying to set up PXE booting (which requires TFTP) on one of my networking that is hiding behind a NAT router.

My question is similar to many others around the 'Net, but all the answers I found applied to CentOS 7 with iptables. I need to do this with CentOS 8 with firewalld and nft as the backend.

Unable to NAT TFTP traffic because iptables is not forwarding the return connection to the client despite TFTP helper creating an expectation https://unix.stackexchange.com/questions/579508/iptables-rules-to-forward-tftp-via-nat

Here is my simplified network diagram:

     Outside NAT               Inside NAT
10.0.10.10  10.0.10.11->192.168.1.1  192.168.1.2
TFTP server --------> NAT ---------> PXE/TFTP client

TFTP is not working. With tcpdump, I see that the RRQ message travels successfully from 192.168.1.2 to 10.0.10.10. The response arrives at the router, but is not properly NATed to arrive at the client.

I tried it with both settings for sysctl net.netfilter.nf_contrack_helper (rebooted after changing the setting):

# sysctl -a | grep conntrack_helper
net.netfilter.nf_conntrack_helper = 0

With nf_contrack_helper=0:

tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

Initial RRQ:

14:02:27.842563 IP (tos 0x0, ttl 64, id 64642, offset 0, flags [DF], proto UDP (17), length 54)
    192.168.1.2.36799 > 10.0.10.10.69: [udp sum ok]  26 RRQ "grub2/grubx64.efi" octet

Initial RRQ after NAT:

14:02:27.842619 IP (tos 0x0, ttl 63, id 64642, offset 0, flags [DF], proto UDP (17), length 54)
    10.0.10.11.36799 > 10.0.10.10.69: [udp sum ok]  26 RRQ "grub2/grubx64.efi" octet

Response from TFTP server to NAT router:

14:02:27.857924 IP (tos 0x0, ttl 63, id 60000, offset 0, flags [none], proto UDP (17), length 544)
    10.0.10.10.60702 > 10.0.10.11.36799: [udp sum ok] UDP, length 516

(repeated several times until timeout)

With nf_contrack_helper=1, the outgoing packet is not even NATed at all:

tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

Initial RRQ:

14:02:27.842563 IP (tos 0x0, ttl 64, id 64642, offset 0, flags [DF], proto UDP (17), length 54)
    192.168.1.2.36799 > 10.0.10.10.69: [udp sum ok]  26 RRQ "grub2/grubx64.efi" octet

(repeated several times until timeout)

The nf_*_tftp helpers are both loaded (regardless of the nf_contrack_helper setting):

# lsmod | grep tftp
nf_nat_tftp            16384  0
nf_conntrack_tftp      16384  3 nf_nat_tftp
nf_nat                 36864  3 nf_nat_ipv6,nf_nat_ipv4,nf_nat_tftp
nf_conntrack          155648  10 nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,nf_conntrack_tftp,nft_ct,nf_nat_ipv6,nf_nat_ipv4,nf_nat_tftp,nft_masq,nft_masq_ipv4

One of the article linked above suggests the following using iptables (which makes sense):

iptables -A PREROUTING -t raw -p udp --dport 69 -s 192.168.11.0/24 -d 172.16.0.0/16 -j CT --helper tftp

How would I do the equivalent with firewalld with an nft backend.

Update:

The firewalld configuration is fairly complex, so I'm only adding the relevant zones:

The external zone:

<?xml version="1.0" encoding="utf-8"?>
<zone>
  <source address="10.0.10.0/24"/>
  <service name="tftp-client"/>
  <service name="ssh"/>
  <masquerade/>
</zone>

And the internal zone:

<?xml version="1.0" encoding="utf-8"?>
<zone>
  <source address="192.168.1.0/24"/>
  <service name="dhcp"/>
  <service name="ssh"/>
  <service name="dns"/>
  <service name="tftp"/>
  <masquerade/>
</zone>

Note: the Masquerade on the internal zone is a mistake. I removed it, but the behavior is not changing.

Zone drifting is disabled.

Update 2:

To answer a request from a commenter:

DHCP configuration

The DHCP server is running on the same system as the NAT router (192.168.1.1 in the network diagram). It is standard ISC DHCP, handing out IP addresses (as fixed-address; there is no pool involved), mask, gateway, DNS server, etc., as well as the PXE Boot next-server and filename options.

All this obviously works. tcpdump shows that the client sends the correct RRQ packet to the server.

The response arrives back at the NAT router, but does not get sent to the behind-the-NAT side.

Details about how TFTP works and how it breaks with NAT

If you understand the TFTP protocol, it is fairly clear what is happening; I just do not know how to handle it with firewalld/nft/CentOS 8.

Fundamentally, the problem is that the TFTP protocol uses UDP ports in a non-standard way. In "standard" UDP-based protocols such as DNS, the response comes from the same port that the server listens on.

Request: client:54321 -> server:53
Response: server:53 -> client:54321

(where 54321 can be any random ephemeral port number picked by the client).

NAT matches up those IP addresses and ports to identify which response belongs to which request.

TFTP does it differently; the responses do not come from port 69, but some other random port.

Request (RRQ): client54321 -> server:69
Response (Data): server:12345 -> client:54321

Where 54321 is again a random ephemeral port the client chooses, and 12345 is a random ephemeral port the server chooses.

As a result, standard NAT behavior will not find a connection matching an origin server:12345, and drops the packet.

The solution to this problem involves using a helper - the nf_nat_tftp kernel module that understands this quirk.

I just have not been able to figure out how to implement this using CentOS 8, nftables and firewalld.

An answer that uses nftables is perfectly acceptable for me, as long as it does not break any firewalld rules.

Kevin Keane
  • 860
  • 1
  • 8
  • 13
  • No, masquerade for other protocols, such as ICMP, HTTP, SSH, etc. works just fine as it is. The problem is specific to TFTP. It also wouldn't make sense to add interface information because there can (and in my case are) multiple zones on the same external interface. – Kevin Keane Nov 11 '20 at 00:33
  • Thanks, I learned something today. OK, uh, I don't see a port forwarding declared here. Is the TFTP server on the same machine as the NAT router? Your post seems to indicate that it is not. – Michael Hampton Nov 11 '20 at 00:36
  • You are correct, the TFTP server is on a different system altogether. There shouldn't be any port forwarding because this is an outbound connection. Port forwarding is generally used for inbound connections. – Kevin Keane Nov 11 '20 at 00:38
  • You're right again, I read that backward. – Michael Hampton Nov 11 '20 at 00:47
  • No problem. It's not trivial to think through, and it took me a while to even write up the question in a way that is halfway sensible, so it is expected that there is some confusion. – Kevin Keane Nov 11 '20 at 00:50
  • Is an answer using only nftables (but trying to not disrupt firewalld) fine? Also can you give more details on the DHCP configuration that is handling the TFTP server address? Is the NAT router also the DHCP server (not that it matters much)? Could you detail what is the client receiving: as address, as router (if there is any) and as tftp server address (as set with siaddr / dhcpd's *next-server* statement)? What is the role of the 172.16.0.0/16 network? It appears in a rule but it's not described. Should that be the DHCP server (acting as TFTP server) instead? – A.B Nov 11 '20 at 13:20
  • An answer with just nftable is acceptable, as long as it is compatible with firewalld. Thanks! Yes, the NAT router is also the DHCP server. ISC DHCP, relatively plain vanilla configuration. The client receives the standard: IP address, mask, gateway, plus the PXE boot options. It is enough for the client to send the correct TFTP RRQ record to the correct TFTP server. The response to that RRQ is not NATed correctly. – Kevin Keane Nov 12 '20 at 03:28
  • @A.B I missed your question about the 172.16.0.0/16 network. This is not my network; the rule in question was quoted from one of the SE articles I had cited. That is an iptables rules for CentOS 7 that obviously does not apply to me. – Kevin Keane Nov 12 '20 at 04:45
  • The thing is, the current version of firewalld automatically loads the helper. You shouldn't need to do anything further, as with older versions. That it's not working indicates something unusual is going on. – Michael Hampton Nov 12 '20 at 05:47
  • @MichaelHampton Yes, I thought the same thing. Figuring that out is what I'm looking for help with. BTW, the helpers are by default disabled even when loaded. That is what the sysctl net.netfilter.nf_conntrack_helper does. But enabling helpers didn't solve the problem. – Kevin Keane Nov 12 '20 at 16:54
  • I found what is not working in firewalld: it doesn't handle helper-tftp-udp in the forward chains, but in the input chains, which will never match. Maybe you know how to configure firewalld so TFTP is handled as a routed service rather than a local service? – A.B Nov 12 '20 at 23:29

1 Answers1

4

Reason it's not working

It appears firewalld might be geared to handle firewalling local services, rather than routed services.

So the tftp settings will add in the end these nft rules when firewalld has been configured (on CentOS 8) with the zones files in OP (just showing the rules, not the whole ruleset here):

table inet firewalld {
    chain filter_IN_external_allow {
        udp dport 69 ct helper set "helper-tftp-udp"
    }
    chain filter_IN_internal_allow {
        udp dport 69 ct helper set "helper-tftp-udp"
        udp dport 69 ct state { new, untracked } accept
    }
}

Those rules will never match and are thus useless: they are in the input path, not in the forward path.

With the running firewall, these (blindly copied) rules added at the right place: in the forward path, will make TFTP work:

nft insert rule inet firewalld filter_FWDI_internal_allow udp dport 69 ct helper set "helper-tftp-udp"
nft add rule inet firewalld filter_FWDI_internal_allow index 0 udp dport 69 ct state '{ new, untracked }' accept

So in the end a so-called direct option would still be an option so everything is stored in firewalld's configuration. Alas the documentation is a bit misleading:

Warning: Direct rules behavior is different depending on the value of FirewallBackend. See CAVEATS in firewalld.direct(5).

Not reading carefully one would think with FirewallBackend=nftables that it would behave differently by accepting nftables rules, but that's not the case:

# firewall-cmd --version
0.8.0

# firewall-cmd --direct --add-rule inet firewalld filter_FWDI_internal_allow 0 'udp dport 69 ct helper set "helper-tftp-udp" ct state new accept'
Error: INVALID_IPV: invalid argument: inet (choose from 'ipv4', 'ipv6', 'eb')

No need to test much more, this "feature" is documented there:

https://bugzilla.redhat.com/show_bug.cgi?id=1692964

and there:

https://github.com/firewalld/firewalld/issues/555

Direct rules still use iptables with the nftables backend. The CAVEAT is about the order of rules evaluation.

Handle this in an other table

I don't see the point anymore of doing this with firewall-cmd, which will add iptables rules along nftables rules. It just becomes cleaner to add an independent table. It'll just be in the ip family since filters for the specific IPv4 networks will also be added (inet would also be fine).

handletftp.nft (to be loaded with nft -f handletftp.nft):

table ip handletftp
delete table ip handletftp

table ip handletftp {
    ct helper helper-tftp {
        type "tftp" protocol udp
    }

    chain sethelper {
        type filter hook forward priority 0; policy accept;
        ip saddr 192.168.1.0/24 ip daddr 10.0.10.10 udp dport 69 ct helper set "helper-tftp"
    }
}

As the table is different and the ruleset is never flushed, but instead the specific table is (atomically) deleted and recreated, this doesn't affect firewalld nor firewalld will affect it.

The priority doesn't matter much: that this chain is traversed before or after firewalld's chains won't change the fate of the packet (still in the hands of firewalld). Whatever the order, if the packet is accepted by firewalld it will also have activated the helper for this flow.

If you choose to use the nftables service to load this table, you'll have to edit it (eg: systemctl edit --full nftables), because beside loading some probably inadequate default rules, it will flush all rules on stop or reload, disrupting firewalld.

Now, a TFTP transfer will work and activate the specific helper, as can be checked by running two conntrack commands during the transfer:

# conntrack -E & conntrack -E expect
[1] 3635
    [NEW] 300 proto=17 src=10.0.10.10 dst=10.0.10.11 sport=0 dport=56597 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.1.2 master-dst=10.0.10.10 sport=56597 dport=69 class=0 helper=tftp
    [NEW] udp      17 29 src=192.168.1.2 dst=10.0.10.10 sport=56597 dport=69 [UNREPLIED] src=10.0.10.10 dst=10.0.10.11 sport=69 dport=56597 helper=tftp
[DESTROY] 299 proto=17 src=10.0.10.10 dst=10.0.10.11 sport=0 dport=56597 mask-src=255.255.255.255 mask-dst=255.255.255.255 sport=0 dport=65535 master-src=192.168.1.2 master-dst=10.0.10.10 sport=56597 dport=69 class=0 helper=tftp
    [NEW] udp      17 30 src=10.0.10.10 dst=10.0.10.11 sport=42032 dport=56597 [UNREPLIED] src=192.168.1.2 dst=10.0.10.10 sport=56597 dport=42032
 [UPDATE] udp      17 30 src=10.0.10.10 dst=10.0.10.11 sport=42032 dport=56597 src=192.168.1.2 dst=10.0.10.10 sport=56597 dport=42032

The 3rd NEW entry in the example above is actually tagged as RELATED (that's the whole role of the tftp helper: expect a certain type of packet to get it seen as related) which will be accepted by the firewall.

A.B
  • 9,037
  • 2
  • 19
  • 37
  • Of course if somebody knows how to have firewalld add the specific tftp rules in the forward chain, that would be an improvement over this answer. – A.B Nov 13 '20 at 01:28
  • Thank you, thank you! I will give this a try. – Kevin Keane Nov 13 '20 at 21:26
  • And it is working! Thank you so much. – Kevin Keane Nov 14 '20 at 02:24
  • It works really well, but on reboot, the rules are lost. I just posted a follow-up question about that. https://serverfault.com/questions/1046179/rhel-centos-now-to-add-nftable-rules-to-firewalld-on-system-boot – Kevin Keane Dec 15 '20 at 01:43