3
I have something really weird going on that I can't seem to find any reference to after a lot of googling. I seem to have no iptables. Not that the chains are flushed or that they are all ACCEPT rules or something, the tables themselves don't seem to exist. Here is what I mean:
The story is, my docker stopped working at some point in the last few months and I finally got around to fixing it. The error was being caused by the following command:
$ iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN
iptables: No chain/target/match by that name.
Which docker runs as part of its startup and which I tried to run manually to debug.
So then I started messing around trying to add different chains and rules in different places, and everything was giving that error. So finally I tried to just list everything
$sudo iptables -S
iptables: No chain/target/match by that name.
$ sudo iptables -L
iptables: No chain/target/match by that name.
$ sudo iptables --list
iptables: No chain/target/match by that name.
nothing. So I tried to look at each of the tables
# iptables -vL -t filter
iptables: No chain/target/match by that name.
# iptables -vL -t nat
iptables: No chain/target/match by that name.
# iptables -vL -t mangle
iptables: No chain/target/match by that name.
# iptables -vL -t raw
iptables: No chain/target/match by that name.
# iptables -vL -t security
iptables: No chain/target/match by that name.
More nothing, it's like the actual tables themselves are gone. Even something as simple as
# iptables -P INPUT ACCEPT
iptables: Bad built-in chain name.
doesn't work.
Has anyone seen this before? Is there any way to get the tables back?
My system is Ubuntu 18.10 with Kernel 5.1.8
Updates
I have since added all the iptables modules to my /etc/modules
and rebuilt the initramfs. The modules are now loaded on boot but it didn't solve the problem.
I found that the iptables-save
command does not error, but it also only prints the following:
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*nat
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*mangle
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*raw
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*security
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*filter
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
I also found that ip6tables appears to be working normally, its only iptables that is broken.
Next I tried running some of the iptables commands in verbose mode.
# iptables -S -vv
libiptc vlibxtables.so.12. 0 bytes.
Table `filter'
Hooks: pre/in/fwd/out/post = 7f68/9f6085dd/5616/9f60a8e0/5616
Underflows: pre/in/fwd/out/post = 36e4540/7fff/36e48e8/7fff/0
iptables: No chain/target/match by that name.
# iptables -N DOCKER-ISOLATION-STAGE-1 -vv
In verbose mode this commant doesn't complete, the output is huge. I tried dumping it to a file but I killed it when that file reached 8.5GB in size. The output is all repitions of the following pattern:
libiptc vlibxtables.so.12. 1032595540 bytes.
Table `filter'
Hooks: pre/in/fwd/out/post = 7ffe/92c0b5dd/55a7/92c0d8e0/55a7
Underflows: pre/in/fwd/out/post = 3d8c10f0/7ffe/3d8c1498/7ffe/3d8c2854
Entry 0 (0):
SRC IP: 0.0.0.0/0.0.0.0
DST IP: 0.0.0.0/0.0.0.0
Interface: `'/................to `'/................
Protocol: 0
Flags: 00
Invflags: 00
Counters: 0 packets, 0 bytes
Cache: 00000000
Target name: `' [0]
verdict=0
Entry 0 (0):
SRC IP: 0.0.0.0/0.0.0.0
DST IP: 0.0.0.0/0.0.0.0
Interface: `'/................to `'/................
Protocol: 0
Flags: 00
Invflags: 00
Counters: 0 packets, 0 bytes
Cache: 00000000
Target name: `' [0]
verdict=0
Hopefully this makes sense to someone, it's meaningless to me.
1What happens if you reboot? Does your kernel have
ip_tables
built-in or as a module? If it is a module, doeslsmod
show it is present? – dirkt – 2019-06-10T06:06:15.983Good call, they're modules and they weren't loaded automatically. I
modprobe
dip_tables
as well as the all the table modules (iptables_filter
, etc) but I still have the same errors – Max Ehrlich – 2019-06-10T12:35:06.4471It's really strange those modules don't get loaded automatically. Did you reboot? Any error messages in
dmesg
after reboot? – dirkt – 2019-06-10T14:36:29.860Yes I rebooted and also upgraded to kernel 5.1.8, nothing unusual in dmesg – Max Ehrlich – 2019-06-10T15:04:45.580
@dirkt I updated the post with some more information I collected, maybe you can make sense of it, I couldn't – Max Ehrlich – 2019-06-11T21:44:16.350
1What the command
iptables
does is to communicate with the kernel, and it looks like this communication goes wrong, because something in the kernel is not working as expected. In particular, normally the required modules would autoload. You do use a stock kernel, a normal Ubuntu system, no custom security restrictions etc.? Nothing funny in the kernel module management? Just in case: Is the package that contains theiptables
command current and matches the kernel? (Though i"ve never seen difficulties in this respect before). – dirkt – 2019-06-12T07:07:56.753I actually do use a custom kernel, but the config is (supposed to be) copied from the mainline ubuntu config, and it just adds ACS override for pcie devices so it shouldn't be causing a problem. Just in case, the config is here. Can I build just the iptables binary from source or does it come with the kernel?
– Max Ehrlich – 2019-06-12T12:25:14.107Tried building latest iptables binaries from the source and that also has the same error message – Max Ehrlich – 2019-06-12T12:42:40.420
1Try to fix it by reinstalling the docker-engine:
apt-get remove docker-engine
followed byapt-get install docker-engine
. – harrymc – 2019-06-13T10:01:07.510@harrymc I tried many times, it's not a docker problem, docker fails because it cant make it's iptables rules because iptables is broken – Max Ehrlich – 2019-06-13T12:31:41.060
1
Are you using the latest docker version? You are using DOCKER-ISOLATION-STAGE-1, but it seems that the namespace is now DOCKER. An example iptables save is available here. (I'm trying to help but I'm not using docker.)
– harrymc – 2019-06-13T13:32:01.510I tried the restore command from your second link just in case but it has the same error. It isn't a docker problem it's an iptables problem – Max Ehrlich – 2019-06-13T20:49:37.010