Understanding packets dropped by iptables from puppetmaster

Question

I am seeing packets getting dropped during my puppet client runs on a semi regular basis. And I do not understand what is causing this. I should mention that Puppet is managing the firewall rules via puppetlabs/firewall module. But the rules do not appear to be getting changed. By that statement, puppet debugging doesn't show that the rules are getting removed and then added, packet counts continue from before the puppet agent run, and viewing the rules during the puppet agent run does not show any rule change during the run.

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            /* 000 accept all icmp */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* 001 accept all to lo interface */
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* 002 accept related established rules */ state RELATED,ESTABLISHED
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            multiport ports 22 /* 100 allow ssh access */
ACCEPT     tcp  --  172.xx.xx.xx         0.0.0.0/0            multiport ports 5666 /* 101 allow nrpe access from 172.xx.xx.xx */
ACCEPT     tcp  --  69.xx.xx.xx          0.0.0.0/0            multiport ports 5666 /* 101 allow nrpe access from 69.xx.xx.xx */
ACCEPT     tcp  --  192.168.116.0/24     192.168.116.0/24     multiport ports 3306 /* 300 allow MySQL BackNet */
LOG        tcp  --  0.0.0.0/0            0.0.0.0/0            /* 301 log all other BackNet requests */ LOG flags 0 level 6 prefix "[IPTABLES INPUT] dropped "
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            /* 302 drop all other BackNet requests */
LOG        tcp  --  0.0.0.0/0            0.0.0.0/0            /* 998 log all other requests */ LOG flags 0 level 6 prefix "[IPTABLES INPUT] dropped "
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            /* 999 drop all other requests */

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            /* 002 accept related established rules OUT */ state RELATED,ESTABLISHED
ACCEPT     tcp  --  172.31.100.0/23      172.31.100.0/23      multiport ports 389,636 /* 351 allow LDAPNet OUT */
ACCEPT     tcp  --  192.168.116.0/24     192.168.116.0/24     multiport ports 3306 /* 351 allow MySQL BackNet OUT */
LOG        tcp  --  0.0.0.0/0            0.0.0.0/0            /* 352 drop all other BackNet requests OUT */ LOG flags 0 level 6 prefix "[IPTABLES OUTPUT] dropped "
LOG        tcp  --  0.0.0.0/0            0.0.0.0/0            /* 352 drop all other LDAPNet requests OUT */ LOG flags 0 level 6 prefix "[IPTABLES OUTPUT] dropped "
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            /* 353 drop all other BackNet requests OUT */
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            /* 353 drop all other LDAPNet requests OUT */

And I am seeing dropped packets on the puppet client, from the puppet master. logged as:

Aug 13 14:22:19 int-vs-repo kernel: [91935.669812] [IPTABLES INPUT] dropped IN=eth0 OUT= MAC=00:50:56:bc:0c:4c:00:e0:b6:00:44:f6:08:00 SRC=PuppetMasterIP DST=Local_IP LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=48739 DF PROTO=TCP SPT=8140 DPT=40069 WINDOW=749 RES=0x00 ACK URGP=0
Aug 13 14:22:19 int-vs-repo kernel: [91935.670108] [IPTABLES INPUT] dropped IN=eth0 OUT= MAC=00:50:56:bc:0c:4c:00:e0:b6:00:44:f6:08:00 SRC=PuppetMasterIP DST=Local_IP LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=48740 DF PROTO=TCP SPT=8140 DPT=40069 WINDOW=749 RES=0x00 ACK URGP=0
Aug 13 14:22:19 int-vs-repo kernel: [91935.670134] [IPTABLES INPUT] dropped IN=eth0 OUT= MAC=00:50:56:bc:0c:4c:00:e0:b6:00:44:f6:08:00 SRC=PuppetMasterIP DST=Local_IP LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=48741 DF PROTO=TCP SPT=8140 DPT=40069 WINDOW=749 RES=0x00 ACK URGP=0
Aug 13 14:22:19 int-vs-repo kernel: [91935.670409] [IPTABLES INPUT] dropped IN=eth0 OUT= MAC=00:50:56:bc:0c:4c:00:e0:b6:00:44:f6:08:00 SRC=PuppetMasterIP DST=Local_IP LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=48742 DF PROTO=TCP SPT=8140 DPT=40069 WINDOW=749 RES=0x00 ACK URGP=0

I am not overly concerned about these specific packets, but plan on moving more machines to have centrally controlled iptables rules and am concerned that this is an indicator of a problem that can't be ignored. It appears as if all the packets dropped like this are ACK's and their ID increments by one, i.e. it appears to me like a connection is getting lost in the middle, and that the ESTABLISHED,RELATED rule should allow these packets.

Aug 14 21:52:19 int-vs-repo kernel: [205448.464080] [IPTABLES INPUT 998] dropped IN=eth0 OUT= MAC=00:50:56:bc:0c:4c:00:e0:b6:00:44:f6:08:00 SRC=PuppetMasterIP DST=PuppetClientIP LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40139 DF PROTO=TCP SPT=8140 DPT=52532 WINDOW=726 RES=0x00 ACK URGP=0
Aug 14 21:52:19 int-vs-repo kernel: [205448.464110] [IPTABLES INPUT 998] dropped IN=eth0 OUT= MAC=00:50:56:bc:0c:4c:00:e0:b6:00:44:f6:08:00 SRC=PuppetMasterIP DST=PuppetAgentIP LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=40140 DF PROTO=TCP SPT=8140 DPT=52532 WINDOW=726 RES=0x00 ACK URGP=0

Which machine is all this happening on, the puppetmaster or the puppet client? Also, having unique log prefixes would make life *significantly* easier in tracking down which DROP rule is being hit. — womble, Aug 14 '15 at 00:14
These drops are on the puppet client, sorry if I did not make that clear, They are also dropped by the line with the comment /* 999 drop all other requests */, this had been seen by packet counts. But I will adjust my log prefixes and verify this. And yes, that would have made things clearer in the example, appologies. — Corvar, Aug 14 '15 at 13:58

Understanding packets dropped by iptables from puppetmaster

0 Answers0