I Have No IPTables

3

I have something really weird going on that I can't seem to find any reference to after a lot of googling. I seem to have no iptables. Not that the chains are flushed or that they are all ACCEPT rules or something, the tables themselves don't seem to exist. Here is what I mean:

The story is, my docker stopped working at some point in the last few months and I finally got around to fixing it. The error was being caused by the following command:

$ iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN
iptables: No chain/target/match by that name.

Which docker runs as part of its startup and which I tried to run manually to debug.

So then I started messing around trying to add different chains and rules in different places, and everything was giving that error. So finally I tried to just list everything

$sudo iptables -S 
iptables: No chain/target/match by that name.
$ sudo iptables -L
iptables: No chain/target/match by that name.
$ sudo iptables --list
iptables: No chain/target/match by that name.

nothing. So I tried to look at each of the tables

# iptables -vL -t filter
iptables: No chain/target/match by that name.
# iptables -vL -t nat
iptables: No chain/target/match by that name.
# iptables -vL -t mangle
iptables: No chain/target/match by that name.
# iptables -vL -t raw
iptables: No chain/target/match by that name.
# iptables -vL -t security
iptables: No chain/target/match by that name.

More nothing, it's like the actual tables themselves are gone. Even something as simple as

# iptables -P INPUT ACCEPT
iptables: Bad built-in chain name.

doesn't work.

Has anyone seen this before? Is there any way to get the tables back?

My system is Ubuntu 18.10 with Kernel 5.1.8

Updates

I have since added all the iptables modules to my /etc/modules and rebuilt the initramfs. The modules are now loaded on boot but it didn't solve the problem.

I found that the iptables-save command does not error, but it also only prints the following:

# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*nat
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*mangle
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*raw
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*security
COMMIT
# Completed on Tue Jun 11 17:35:52 2019
# Generated by iptables-save v1.6.1 on Tue Jun 11 17:35:52 2019
*filter
COMMIT
# Completed on Tue Jun 11 17:35:52 2019

I also found that ip6tables appears to be working normally, its only iptables that is broken.

Next I tried running some of the iptables commands in verbose mode.

# iptables -S -vv 
libiptc vlibxtables.so.12. 0 bytes.
Table `filter'
Hooks: pre/in/fwd/out/post = 7f68/9f6085dd/5616/9f60a8e0/5616
Underflows: pre/in/fwd/out/post = 36e4540/7fff/36e48e8/7fff/0
iptables: No chain/target/match by that name.
# iptables -N  DOCKER-ISOLATION-STAGE-1 -vv

In verbose mode this commant doesn't complete, the output is huge. I tried dumping it to a file but I killed it when that file reached 8.5GB in size. The output is all repitions of the following pattern:

libiptc vlibxtables.so.12. 1032595540 bytes.
Table `filter'
Hooks: pre/in/fwd/out/post = 7ffe/92c0b5dd/55a7/92c0d8e0/55a7
Underflows: pre/in/fwd/out/post = 3d8c10f0/7ffe/3d8c1498/7ffe/3d8c2854
Entry 0 (0):
SRC IP: 0.0.0.0/0.0.0.0
DST IP: 0.0.0.0/0.0.0.0
Interface: `'/................to `'/................
Protocol: 0
Flags: 00
Invflags: 00
Counters: 0 packets, 0 bytes
Cache: 00000000
Target name: `' [0]
verdict=0

Entry 0 (0):
SRC IP: 0.0.0.0/0.0.0.0
DST IP: 0.0.0.0/0.0.0.0
Interface: `'/................to `'/................
Protocol: 0
Flags: 00
Invflags: 00
Counters: 0 packets, 0 bytes
Cache: 00000000
Target name: `' [0]
verdict=0

Hopefully this makes sense to someone, it's meaningless to me.

Max Ehrlich

Posted 2019-06-10T02:52:01.077

Reputation: 66

1What happens if you reboot? Does your kernel have ip_tables built-in or as a module? If it is a module, does lsmod show it is present? – dirkt – 2019-06-10T06:06:15.983

Good call, they're modules and they weren't loaded automatically. I modprobed ip_tables as well as the all the table modules (iptables_filter, etc) but I still have the same errors – Max Ehrlich – 2019-06-10T12:35:06.447

1It's really strange those modules don't get loaded automatically. Did you reboot? Any error messages in dmesg after reboot? – dirkt – 2019-06-10T14:36:29.860

Yes I rebooted and also upgraded to kernel 5.1.8, nothing unusual in dmesg – Max Ehrlich – 2019-06-10T15:04:45.580

@dirkt I updated the post with some more information I collected, maybe you can make sense of it, I couldn't – Max Ehrlich – 2019-06-11T21:44:16.350

1What the command iptables does is to communicate with the kernel, and it looks like this communication goes wrong, because something in the kernel is not working as expected. In particular, normally the required modules would autoload. You do use a stock kernel, a normal Ubuntu system, no custom security restrictions etc.? Nothing funny in the kernel module management? Just in case: Is the package that contains the iptables command current and matches the kernel? (Though i"ve never seen difficulties in this respect before). – dirkt – 2019-06-12T07:07:56.753

I actually do use a custom kernel, but the config is (supposed to be) copied from the mainline ubuntu config, and it just adds ACS override for pcie devices so it shouldn't be causing a problem. Just in case, the config is here. Can I build just the iptables binary from source or does it come with the kernel?

– Max Ehrlich – 2019-06-12T12:25:14.107

Tried building latest iptables binaries from the source and that also has the same error message – Max Ehrlich – 2019-06-12T12:42:40.420

1Try to fix it by reinstalling the docker-engine: apt-get remove docker-engine followed by apt-get install docker-engine. – harrymc – 2019-06-13T10:01:07.510

@harrymc I tried many times, it's not a docker problem, docker fails because it cant make it's iptables rules because iptables is broken – Max Ehrlich – 2019-06-13T12:31:41.060

1

Are you using the latest docker version? You are using DOCKER-ISOLATION-STAGE-1, but it seems that the namespace is now DOCKER. An example iptables save is available here. (I'm trying to help but I'm not using docker.)

– harrymc – 2019-06-13T13:32:01.510

I tried the restore command from your second link just in case but it has the same error. It isn't a docker problem it's an iptables problem – Max Ehrlich – 2019-06-13T20:49:37.010

Answers

1

This turned out to be a kernel config problem. Usually when I apply my patch a build a new kernel, I copy the config I was using from the previous version (in hindsight maybe a bad idea). Once I merged my config with the latest Ubuntu kernel config and rebuilt the kernel, my iptables output looked normal again.

Max Ehrlich

Posted 2019-06-10T02:52:01.077

Reputation: 66

2

Searching on internet i found that is possible to restore iptables on Linux using the following command

iptables-restore < /root/working.iptables.rules

However there are some technical guides which suggest that if using kubernetes proxy iptables rules lost after restarting iptables+node.

The following files provides useful information about it:

https://www.cyberciti.biz/faq/how-to-save-restore-iptables-firewall-config-ubuntu/

https://github.com/openshift/origin/issues/1922

https://www.linuxquestions.org/questions/linux-security-4/iptables-no-chain-tarjet-match-by-that-name-941406/

I hope that can help!

user1039968

Posted 2019-06-10T02:52:01.077

Reputation:

1Thanks a lot for the reply, unfortunately the restore command would have required I saved the rules using iptables-save (creating the file /root/working.iptables.rules) when they were still working. The other suggestions unfortunately also had no effect. – Max Ehrlich – 2019-06-12T00:23:18.827

1

Have you tried manually reinstalling iptables?

./configure --prefix=/usr      \
            --sbindir=/sbin    \
            --disable-nftables \
            --enable-libipq    \
            --with-xtlibdir=/lib/xtables &&
make
make install &&
ln -sfv ../../sbin/xtables-legacy-multi /usr/bin/iptables-xml &&

for file in ip4tc ip6tc ipq iptc xtables
do
  mv -v /usr/lib/lib${file}.so.* /lib &&
  ln -sfv ../../lib/$(readlink /usr/lib/lib${file}.so) /usr/lib/lib${file}.so
done
make install-iptables

Drawn from here.

johnbchron

Posted 2019-06-10T02:52:01.077

Reputation: 110

yes I tried using built-from-source iptables and it gives the same errors – Max Ehrlich – 2019-06-13T18:35:54.607

okay well honestly I don't have anything left to recommend to you than a new clean install – johnbchron – 2019-06-13T18:40:31.163