Debugger for Iptables

Question

I'm looking for an easy way to follow a packet through the iptables rules. This is not so much about logging, because I don't want to log all traffic (and I only want to have LOG targets for very few rules).

Something like Wireshark for Iptables. Or maybe even something similar to a debugger for a programming language.

Thanks Chris

Note: It doesn't have to be a fancy GUI tool. But it must do more than just showing a package counter or so.

Update: It almost looks as if we can't find anything that provides the functionality that is asked for. In that case: Let's at least find a good technique that's based on iptables logging - which can be easily turned on and off, and doesn't require to write iptables rules redundantly (having to write the same rule for -j LOG and -j ...)

score 88 · Answer 1 · answered Mar 25 '10 at 05:53

If you have a recent enough kernel and version of iptables you can use the TRACE target (Seems to be builtin on at least Debian 5.0). You should set the conditions of your trace to be as specific as possible and disable any TRACE rules when you are not debugging because it does spew a lot of information to the logs.

TRACE
This target marks packes so that the kernel will log every rule which match the packets as those traverse the tables, chains, rules. (The ipt_LOG or ip6t_LOG module is required for the logging.) The packets are logged with the string prefix: "TRACE: tablename:chainname:type:rulenum " where type can be "rule" for plain rule, "return" for implicit rule at the end of a user defined chain and "policy" for the policy of the built in chains. It can only be used in the raw table.

If you added rules like this

iptables -t raw -A PREROUTING -p tcp --destination 192.168.0.0/24 --dport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --destination 192.168.0.0/24 --dport 80 -j TRACE

You will be supplied with output that looks like this.

# cat /var/log/kern.log | grep 'TRACE:'
Mar 24 22:41:52 enterprise kernel: [885386.325658] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.12.152 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=80 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325689] TRACE: mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.12.152 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=80 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325713] TRACE: nat:PREROUTING:rule:1 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.12.152 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=80 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: nat:nat.1:rule:1 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.12.152 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=80 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: mangle:INPUT:policy:1 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.32.10 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=3128 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: filter:INPUT:rule:2 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.32.10 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=3128 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: filter:in_world:rule:1 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.32.10 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=3128 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: filter:in_world_all_c1:return:2 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.32.10 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=3128 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: filter:in_world:rule:2 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.32.10 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=3128 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)
Mar 24 22:41:52 enterprise kernel: [885386.325731] TRACE: filter:in_world_irc_c2:return:2 IN=eth0 OUT= MAC=00:1d:7d:aa:e3:4e:00:04:4b:05:b4:dc:08:00 SRC=192.168.32.18 DST=192.168.32.10 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=30561 DF PROTO=TCP SPT=53054 DPT=3128 SEQ=3653700382 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0 OPT (020405B40103030201010402)

Thanks, this is awesome! It's actually the best answer, I wish I could accept it (it was a bounty question, so the accepted answer is definite). While I can't use it on all of my systems (due to kernel limitations), on some systems I can. This answer deserves upvoting, because it's really close to what I was looking for. — Chris Lercher, Mar 25 '10 at 14:43
I found this feature last night when I was re-reading the iptables man page so I could answer a different question. Seems to be a relatively new feature. No worries about not being able to mark it as accepted. Maybe this will get voted up enough over time to earn me another Populist badge. — Zoredache, Mar 25 '10 at 18:43
This is really the canonical answer for tracing packets in iptables. It's too bad that some recent kernels don't enable it by default. — Peter Grace, Jun 21 '12 at 14:45
How long ago does the kernel support TRACE? I've used with success on CentOS 6.4 but not in CentOS 6.2 — sebelk, Oct 03 '13 at 14:50
I think this answer could be improved by explaining what the modules `ipt_LOG` and `ip6t_LOG` actually do. The module "description" for both simply is: "Xtables: IPv4/IPv6 packet logging". Similar for `ipt_TRACE`: "Xtables: packet flow tracing". — U. Windl, Mar 19 '20 at 10:37

score 10 · Accepted Answer · answered Mar 16 '10 at 12:56

10

I can't think of a direct solution, but I can think of a round about way of tracking a packet.

Log each rule with a log prefix directive (--log-prefix "Rule 34")
Generate a test packet or packet stream with scapy and set the TOS field to something unique
grep the log file output for that TOS setting and see which rules logged it.

answered Mar 16 '10 at 12:56

Haakon

1,305
7
11

Thanks for the idea. Unfortunately, I can't log every rule (On one system, the disk probably wouldn't be fast enough to do that. On another, iptables logging isn't available in the kernel.) – Chris Lercher Mar 16 '10 at 13:13
1

Use a named pipe as the file http://www.softpanorama.org/Logs/Syslog/pipes_in_syslog.shtml However since you can't log in your kernel you are kinda SOL – Haakon Mar 16 '10 at 13:32
Thanks, it probably won't solve my problem, but it's generally nice to know, that piping syslog would be possible - might come in handy at another time! – Chris Lercher Mar 16 '10 at 14:13
One related question about logging: Does iptables handle multiple packets concurrently (so that log entries could be interleaved)? In that case, I think the TOS idea would be an absolute must for a lot of iptables LOG analyses! – Chris Lercher Mar 16 '10 at 19:18
I don't know the answer to that. I expect that each interface would be handled concurrently by iptables at a minimum though. – Haakon Mar 16 '10 at 23:57
Using the TOS is a very good idea. I think I may be creating a script that automates updating the TOS field, as soon as iptables receives a packet - and then something that analyzes the logs. Also something, to turn logging on and off quickly. I currently don't have the time to create something like that, but when an even stronger need arises in the future, I might take the time. – Chris Lercher Mar 21 '10 at 23:38

score 6 · Answer 3 · answered Mar 16 '10 at 16:49

Three answers on one post:

1) Debug by script:

#!/bin/bash
debug() {
    if [ -n "$debug" ]; then
        $@ || echo -e "The command which launched the error:\n$@"
    else
        $@
    fi
}
debug=1
IPTABLES="debug /sbin/iptables"

$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT DROP
....

2) Debug by syslog

From this website :http://www.brandonhutchinson.com/iptables_fw.html

If you want to make a syslog entry of dropped packets, change:

# Drop all other traffic
/sbin/iptables -A INPUT -j DROP

To:

# Create a LOGDROP chain to log and drop packets
/sbin/iptables -N LOGDROP
/sbin/iptables -A LOGDROP -j LOG
/sbin/iptables -A LOGDROP -j DROP

# Drop all other traffic
/sbin/iptables -A INPUT -j LOGDROP


You may also want to configure the --log-level to log dropped packets to a separate file instead of /var/log/messages:

# Drop all other traffic
/sbin/iptables -A INPUT -j LOGDROP --log-level debug


/etc/syslog.conf change:

# Send iptables LOGDROPs to /var/log/iptables
kern.=debug                                             /var/log/iptables

Reload the syslogd service for the change to take effect.
/sbin/service syslog reload

3) No debug, nice iptables edit:

Also this can be helpfull: http://www.fwbuilder.org/

Thanks. Point 1) and 3) don't have much to do with following packets through the iptables rules, but the point about redirecting log entries based on "--log-level" may be helpful, if I finally really have to fall back to logging (in case there's absolutely no other solution). — Chris Lercher, Mar 16 '10 at 19:12

score 2 · Answer 4 · answered May 29 '16 at 17:50

had the same question and found Zoredache pointing to TRACE / ipt_LOG was the solution!

Additionally I found a script which inserts/removes LOG-rules preceding all currently active iptables rules. I tried it out and found it to be a really nice tool. - Output is similar to the TRACE solution - Advantage: it works on the active iptables configuration, no matter where it was loaded from. You can turn logging on/off on the fly! You don't need to modify any firewall-scripts which might have been generated by Firewall Builder or tool whatever you use... - Disadvantage: without modification, the script creates LOG-rules for ALL active rules. Instead, when using TRACE rules, you'll probably restrict logging to addresses/services/connections for which you want to investigate iptables processing now.

Anyhow, I like the aproach :) Kudos to Tony Clayton, have a look: http://lists.netfilter.org/pipermail/netfilter/2003-March/043088.html

Regards, Chris

score 0 · Answer 5 · answered Mar 15 '10 at 12:14

0

I usually use packets and bytes counters to see how rules work and to find what is missing or wrong.

You can view them by "iptables -nvL".

answered Mar 15 '10 at 12:14

Vi.

821
11
19

2

I can see what the author wants, though; if you are trying to test your iptables rules on a busy interface merely watching counters is not going to help a whole lot particularly if the packet is likely to match on several rules and jump around user-defined chains in the process (as is typical when filtering out unwanted IP addresses, and flood protection rules). – PP. Mar 15 '10 at 13:01
@PP: Exactly, you're reading my mind. @Vi: Thanks, this can be helpful in some circumstances, and I've used that sometimes. Now I need something more powerful. – Chris Lercher Mar 15 '10 at 13:33

score -2 · Answer 6 · answered Mar 15 '10 at 12:39

-2

AFAIK an IP packet traverses the rule chain until the first match. So I don't really see what's the problem here. If you have:

rule 1
rule 2
rule 3 LOG

And a packet makes it into the log, it means that rule 3 is the first matching rule.

answered Mar 15 '10 at 12:39

Anonymous

1,540
1
14
18

1

Not true. Packets can match multiple rules, and they do. Unless a rule has an action (like `-j DROP` or `-j ACCEPT`) it will just continue matching further down the chain. – PP. Mar 15 '10 at 12:59

Debugger for Iptables

6 Answers6

Linked

Related