Yesterday I realized that one of my LXC containers on a Proxmox server needed to access github.com, therefore I had to add IPv4 support (really Github?). I added a /29 subnet to the internal network behind the vmbr0 interface. However, routing of packets back to the container does not work. Same is true for connecting from the outside, of course.
Anyways, I did this on several other Proxmox servers in the past already. Never had an issue. Also, IPv6 is working without any issues. And AFAICT I have the same setup on at least 2 other servers with just different IPv4 subnets.
The interface configuration on the Proxmox server is the following:
auto enp35s0 # public interface
iface enp35s0 inet static
address N.M.173.126 # public server address provided by ISP
netmask 26
gateway N.M.173.65
up route add -net N.M.173.64 netmask 255.255.255.192 gw N.M.173.65 dev enp35s0
auto vmbr0
iface vmbr0 inet static
address X.Y.163.145 # subnet assigned by ISP to this server is X.Y.163.144/29
netmask 29
bridge-ports none
bridge-stp off
bridge-fd 0
This is the routing table:
default via N.M.173.65 dev enp35s0
X.Y.163.144/29 dev vmbr0 proto kernel scope link src X.Y.163.145
N.M.173.64/26 via N.M.173.65 dev enp35s0
N.M.173.64/26 dev enp35s0 proto kernel scope link src N.M.173.126
Now, the LXC container has the IP X.Y.163.146 assigned on its eth0 interface:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address X.Y.163.146
netmask 255.255.255.248
gateway X.Y.163.145
Basic networking works. Proxmox (.145) can ping the container (.146), the container can ping the hypervisor.
Now when pinging github.com from the container, I can listen with tcpdump on enp35s0 of the Proxmox server. I see the ICMP request leaving and the reply coming in (Github.com is at 140.82.118.3):
11:54:35.131596 IP X.Y.163.146 > 140.82.118.3: ICMP echo request, id 1204, seq 5, length 64
11:54:35.143779 IP 140.82.118.3 > X.Y.163.146: ICMP echo reply, id 1204, seq 5, length 64
But listening on the vmbr0 interface, I only see the outgoing packet. That means routing back into the internal network is not working.
For testing purposes I stopped the pve-firewall. iptables look like this:
Chain PREROUTING (policy ACCEPT 18825 packets, 6190K bytes)
pkts bytes target prot opt in out source destination
19953 6512K LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
Chain INPUT (policy ACCEPT 16343 packets, 6057K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 200 packets, 16320 bytes)
pkts bytes target prot opt in out source destination
200 16320 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
Chain OUTPUT (policy ACCEPT 14420 packets, 5998K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 14620 packets, 6015K bytes)
pkts bytes target prot opt in out source destination
14659 6027K LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
Everything else is empty and policy is ACCEPT.
IPv4 forwarding is enabled, of course.
cat /proc/sys/net/ipv4/ip_forward
1
Can anyone see what I am missing? Any help to further tackle this would be appreciated.
/edit: I added log rules for all chains:
The corresponding rules:
iptables -t mangle -A PREROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "MANGLE:PREROUTING:IN:GITHUB"
iptables -t mangle -A FORWARD -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "MANGLE:FORWARD:IN:GITHUB"
iptables -t mangle -A POSTROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "MANGLE:POSTROUTING:IN:GITHUB"
iptables -t nat -A PREROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "NAT:PREROUTING:IN:GITHUB"
iptables -t filter -A FORWARD -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "FILTER:FORWARD:IN:GITHUB"
iptables -t nat -A POSTROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "NAT:POSTROUTING:IN:GITHUB"
iptables -t raw -A PREROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "RAW:PREROUTING:IN:GITHUB"
iptables -t mangle -A PREROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:PREROUTING:OUT:PASSBOLT"
iptables -t mangle -A FORWARD -s X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:FORWARD:OUT:PASSBOLT"
iptables -t mangle -A POSTROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:POSTROUTING:OUT:PASSBOLT"
iptables -t nat -A PREROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:PREROUTING:OUT:PASSBOLT"
iptables -t filter -A FORWARD -s X.Y.163.146 -p icmp -j LOG --log-prefix "FILTER:FORWARD:OUT:PASSBOLT"
iptables -t nat -A POSTROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:POSTROUTING:OUT:PASSBOLT"
iptables -t raw -A PREROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "RAW:PREROUTING:OUT:PASSBOLT"
iptables -t mangle -A PREROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:PREROUTING:IN:PASSBOLT"
iptables -t mangle -A FORWARD -d X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:FORWARD:IN:PASSBOLT"
iptables -t mangle -A POSTROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:POSTROUTING:IN:PASSBOLT"
iptables -t nat -A PREROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:PREROUTING:IN:PASSBOLT"
iptables -t filter -A FORWARD -d X.Y.163.146 -p icmp -j LOG --log-prefix "FILTER:FORWARD:IN:PASSBOLT"
iptables -t nat -A POSTROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:POSTROUTING:IN:PASSBOLT"
iptables -t raw -A PREROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "RAW:PREROUTING:IN:PASSBOLT"
The output for one ping is:
Mar 5 15:35:39 proxmox kernel: [18347.757914] RAW:PREROUTING:OUT:LXC IN=fwbr1005i0 OUT= PHYSIN=veth1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758170] MANGLE:PREROUTING:OUT:LXC IN=fwbr1005i0 OUT= PHYSIN=veth1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758426] MANGLE:FORWARD:OUT:LXC IN=fwbr1005i0 OUT=fwbr1005i0 PHYSIN=veth1005i0 PHYSOUT=fwln1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758686] FILTER:FORWARD:OUT:LXC IN=fwbr1005i0 OUT=fwbr1005i0 PHYSIN=veth1005i0 PHYSOUT=fwln1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758963] MANGLE:POSTROUTING:OUT:LXC IN= OUT=fwbr1005i0 PHYSIN=veth1005i0 PHYSOUT=fwln1005i0 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.759189] RAW:PREROUTING:OUT:LXC IN=vmbr0 OUT= PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.759416] MANGLE:PREROUTING:OUT:LXC IN=vmbr0 OUT= PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.759642] MANGLE:FORWARD:OUT:LXC IN=vmbr0 OUT=enp35s0 PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.761541] FILTER:FORWARD:OUT:LXC TIN=vmbr0 OUT=enp35s0 PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.761791] MANGLE:POSTROUTING:OUT:LXC IN= OUT=enp35s0 PHYSIN=fwpr1005p0 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.773606] RAW:PREROUTING:IN:GITHUB IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.773831] RAW:PREROUTING:IN:LXC IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.774051] MANGLE:PREROUTING:IN:GITHUB IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.774288] MANGLE:PREROUTING:IN:LXC IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
So it seems the "MANGLE:PREROUTING:IN:LXC" rule is triggered. But why does the packet not arrive in the FORWARD chain? Just in case I also added a log rule for the INPUT chain. No entry there as well. It seems the packet is dropped without any notice / rule?!
