Using nftables to forward HTTP(s) traffic to VM guest without masquerade

Question

I am currently in the process of setting secure as possible KVM based server. The VM host should be more or less invisible to the internet and only VM guests running single services should be exposed. VM guests should have no internet access (--> no NAT masquerade).

Now I would like to expose HTTP(s) ports of the VM guest 192.168.122.183 to the internet. All existing documentation uses iptables which is almost deprecated. So I prefer a solution using nftables.

According to the documentation, this should be fairly simple, so my nft list ruleset looks accordingly:

table ip nat {
        chain prerouting {
                type nat hook prerouting priority 0; policy accept;
                iif "enp35s0" tcp dport { http, https } dnat to 192.168.122.183
        }
}
table inet filter {
        chain input {
                type filter hook input priority 0; policy accept;
        }

        chain forward {
                type filter hook forward priority 0; policy accept;
        }

        chain output {
                type filter hook output priority 0; policy accept;
        }
}

Also ip forwarding is activated sysctl -a | grep forward:

net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.enp35s0.bc_forwarding = 0
net.ipv4.conf.enp35s0.forwarding = 1
net.ipv4.conf.enp35s0.mc_forwarding = 0
net.ipv4.conf.virbr10.bc_forwarding = 0
net.ipv4.conf.virbr10.forwarding = 1
net.ipv4.conf.virbr10.mc_forwarding = 0
net.ipv4.conf.virbr10-dummy.bc_forwarding = 0
net.ipv4.conf.virbr10-dummy.forwarding = 1
net.ipv4.conf.virbr10-dummy.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0

Accessing the web server from the VM host works perfectly but from the internet I don't get response. What am I missing?

Might be related to this question.

Here is an exerpt from tcpdump from the host. As seen request from the VM host is answered correctly and also the external request is popping up but is never answered. Any ideas why?

23:43:28.048714 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [S], seq 502099991, win 64240, options [mss 1460,sackOK,TS val 3361102460 ecr 0,nop,wscale 7], length 0
23:43:28.048746 IP 192.168.122.183.http > 192.168.122.1.35584: Flags [S.], seq 3142491522, ack 502099992, win 65160, options [mss 1460,sackOK,TS val 519662698 ecr 3361102460,nop,wscale 7], length 0
23:43:28.048801 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 3361102460 ecr 519662698], length 0
23:43:28.048846 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [P.], seq 1:143, ack 1, win 502, options [nop,nop,TS val 3361102460 ecr 519662698], length 142: HTTP: GET / HTTP/1.1
23:43:28.048851 IP 192.168.122.183.http > 192.168.122.1.35584: Flags [.], ack 143, win 508, options [nop,nop,TS val 519662698 ecr 3361102460], length 0
23:43:28.048960 IP 192.168.122.183.http > 192.168.122.1.35584: Flags [P.], seq 1:325, ack 143, win 508, options [nop,nop,TS val 519662698 ecr 3361102460], length 324: HTTP: HTTP/1.1 200 OK
23:43:28.049015 IP 192.168.122.1.35584 > 192.168.122.183.http: Flags [.], ack 325, win 501, options [nop,nop,TS val 3361102460 ecr 519662698], length 0

23:43:45.263545 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:a7:a9.8002, length 35
23:43:45.537492 IP external.dip0.t-ipconnect.de.46866 > 192.168.122.183.http: Flags [S], seq 1122615229, win 64240, options [mss 1452,sackOK,TS val 2536247181 ecr 0,nop,wscale 7], length 0
23:43:47.247528 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:de:a7:a9.8002, length 35
23:43:47.553490 IP external.dip0.t-ipconnect.de.46866 > 192.168.122.183.http: Flags [S], seq 1122615229, win 64240, options [mss 1452,sackOK,TS val 2536249197 ecr 0,nop,wscale 7], length 0

For reference my ip addr output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp35s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a8:a1:59:0e:**:** brd ff:ff:ff:ff:ff:ff
    inet 95.217.***.***/26 brd 95.217.120.127 scope global enp35s0
       valid_lft forever preferred_lft forever
    inet6 2a01:4f9:**:****::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::aaa1:59ff:****:*****/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr10-dummy: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr10 state UNKNOWN group default qlen 1000
    link/ether 52:54:00:de:a7:a9 brd ff:ff:ff:ff:ff:ff
4: virbr10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:de:a7:a9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr10
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fede:a7a9/64 scope link 
       valid_lft forever preferred_lft forever
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr10 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:94:b9:ce brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe94:b9ce/64 scope link 
       valid_lft forever preferred_lft forever

Another question answered that virt-manager is incompatible with nftables as it still uses iptables. But the iptable rules are empty and the iptables modules (especially iptable_nat) are not loaded

# lsmod | grep ip
nft_chain_route_ipv4    16384  0
nft_chain_nat_ipv4     16384  1
ipt_MASQUERADE         16384  0
nf_nat_ipv4            16384  2 ipt_MASQUERADE,nft_chain_nat_ipv4
nf_nat                 36864  2 nft_nat,nf_nat_ipv4
ipt_REJECT             16384  0
nf_reject_ipv4         16384  1 ipt_REJECT
nf_conntrack          172032  5 xt_conntrack,nf_nat,nft_nat,ipt_MASQUERADE,nf_nat_ipv4
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
nf_tables             143360  13 nft_chain_route_ipv4,nft_compat,nft_nat,nft_chain_nat_ipv4,nft_counter,nf_tables_set
ip_tables              28672  0
x_tables               45056  7 xt_conntrack,nft_compat,xt_tcpudp,ipt_MASQUERADE,xt_CHECKSUM,ipt_REJECT,ip_tables

Also, given your open forward chain, your VM does have internet provided it sets your host as gateway. — Ginnungagap, Sep 27 '20 at 22:56
@Ginnungagap It does not have internet access because VM lives in `192.168.122.0/24` private IP address space. — Tero Kilkanen, Sep 28 '20 at 05:25

score 3 · Accepted Answer · answered Sep 28 '20 at 05:29

The following conditions must be fulfilled in order to VM be able to receive connections from outside:

VM has a default route pointing to 192.168.122.1.
Firewall has DNAT rule configured.

In your case, first one is likely missing. This results in VM network stack dropping the SYN-ACK packets sent by the TCP protocol. This happens because the client VM does not know where to forward the response packet.

score -1 · Answer 2 · answered Sep 27 '20 at 22:52

-1

You either need an explicit Source NAT or Masquerading, your VM is likely answering but you're not rewriting the source IP of outgoing packets so packets received by the external client make no sense.

answered Sep 27 '20 at 22:52

Ginnungagap

1,998
8
9

1

Source NAT / Masquerading is needed only for connections initiated by the VM. Connections initiated from outside where destination is the VM are handled correctly with DNAT only. – Tero Kilkanen Sep 28 '20 at 05:23

Using nftables to forward HTTP(s) traffic to VM guest without masquerade

2 Answers2

Linked