1

I've got a simple webserver (apache only) running on CentOS's AMI on AWS inside a VPC. I've got both AWS Security Groups and iptables opened up for port 80, with no source requirements. I then set up an Elastic Load Balancer to direct traffic to the single instance (when everything works, then I'll clone it for multiple load-balanced instances). The instance registered with the load balancer just fine, and is currently listed as 'healthy'

If I pull up the instances's local IP in my browser, I can connect/view pages just fine.

If I pull up the ELB's hostname in my browser, it times out. I check /var/log/messages on the instance, and iptables has dropped the packet. I have no idea why iptables doesn't find a match for these packets but does for the direct connections.

I removed most of the rules in iptables just to make sure nothing interacted weirdly. Here's my current ruleset:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT

-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.40.0.0/16 -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT

-A RH-Firewall-1-INPUT -j LOG --log-prefix "IP DROP NO MATCH: "
-A RH-Firewall-1-INPUT -j DROP
COMMIT

Here's what the dropped packets look like:

Aug 28 21:59:54 ip-10-40-80-11 kernel: IP DROP NO MATCH: IN=eth0 OUT= MAC=0e:57:37:73:05:4e:0e:57:37:40:00:3F:68:21 SRC=10.40.80.179 DST=10.40.80.11 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=2363 DF PROTO=TCP SPT=61866 DPT=80 WINDOW=58 RES=0x00 ACK FIN URGP=0

Out of curiousity, I commented out the --dport 80 line in my iptables config so that any port 80 traffic got blocked; I then visited the instance in my browser by direct IP and looked for the log of dropped packets. I wanted to compare what the dropped packet log would record for 'successful' connections versus my problem with the ELB. Here's what I saw:

Aug 28 22:05:12 ip-10-40-80-11 kernel: IP DROP NO MATCH: IN=eth0 OUT= MAC=0e:57:37:73:05:4e:0e:57:37:40:00:3F:68:21 SRC=10.40.180.110 DST=10.40.80.11 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=2816 DF PROTO=TCP SPT=50995 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0

The only difference (other than the source, which shouldn't matter?) was that the ELB packets ended in ACK FIN while my direct connection ended in SYN ... but I'll admit I don't know enough about the intricacies of TCP to know if this is really making any difference.

What might be the difference that would prevent the ELB HTTP packets from making it through the firewall successfully?

Edit: tcpdump results

# tcpdump -i eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
22:49:38.994125 IP 10.40.80.11.http > 10.40.80.179.62394: Flags [F.], seq 2393122986, ack 3223961569, win 227, options [nop,nop,TS val 116463035 ecr 1792527], length 0
22:49:43.231187 IP 10.40.80.179.62426 > 10.40.80.11.http: Flags [S], seq 1887835523, win 14600, options [mss 1460,sackOK,TS val 1836427 ecr 0,nop,wscale 8], length 0
22:49:43.231267 IP 10.40.80.11.http > 10.40.80.179.62426: Flags [S.], seq 3789128001, ack 1887835524, win 14480, options [mss 1460,sackOK,TS val 116467272 ecr 1836427,nop,wscale 6], length 0
22:49:43.231578 IP 10.40.80.179.62426 > 10.40.80.11.http: Flags [.], ack 1, win 58, options [nop,nop,TS val 1836427 ecr 116467272], length 0
22:49:43.231684 IP 10.40.80.179.62426 > 10.40.80.11.http: Flags [F.], seq 1, ack 1, win 58, options [nop,nop,TS val 1836427 ecr 116467272], length 0
22:49:43.231774 IP 10.40.80.11.http > 10.40.80.179.62426: Flags [F.], seq 1, ack 2, win 227, options [nop,nop,TS val 116467272 ecr 1836427], length 0
22:49:43.232012 IP 10.40.80.179.62426 > 10.40.80.11.http: Flags [.], ack 2, win 58, options [nop,nop,TS val 1836427 ecr 116467272], length 0
22:49:47.632855 IP 10.40.80.179.62427 > 10.40.80.11.http: Flags [S], seq 3419546406, win 14600, options [mss 1460,sackOK,TS val 1837527 ecr 0,nop,wscale 8], length 0
22:49:47.632929 IP 10.40.80.11.http > 10.40.80.179.62427: Flags [S.], seq 1724369749, ack 3419546407, win 14480, options [mss 1460,sackOK,TS val 116471673 ecr 1837527,nop,wscale 6], length 0
22:49:47.632942 IP 10.40.80.179.62416 > 10.40.80.11.http: Flags [F.], seq 3666215037, ack 243876755, win 58, options [nop,nop,TS val 1837527 ecr 116411675], length 0
22:49:47.633055 IP 10.40.80.11.http > 10.40.80.179.62416: Flags [F.], seq 1, ack 1, win 227, options [nop,nop,TS val 116471673 ecr 1837527], length 0
22:49:47.633209 IP 10.40.80.179.62427 > 10.40.80.11.http: Flags [.], ack 1, win 58, options [nop,nop,TS val 1837527 ecr 116471673], length 0
22:49:47.633282 IP 10.40.80.179.62416 > 10.40.80.11.http: Flags [.], ack 2, win 58, options [nop,nop,TS val 1837527 ecr 116471673], length 0
Peter
  • 11
  • 2
  • Can you run a `tcpdump` to capture the connection? The fact that there's `ACK FIN` packets means that it's getting a connection at least partially established successfully, so I think we're missing part of what's going on. – Shane Madden Aug 29 '13 at 04:39
  • tcpdump edited into question. Let me know if those arguments give you the info you were hoping for. – Peter Aug 29 '13 at 04:51

2 Answers2

0

It's likely that iptables doesn't see the ACK/FIN packet as a new connection.
Since the port 80 rule specifies that connections are only allowed if the connection is a new connection, it will drop any connections that aren't new connections.

what happens if you remove -m state --state NEW from the port 80 rule and test ?

Lawrence
  • 380
  • 2
  • 10
  • Interesting... no logging, but still a timeout in the browser... Maybe I need to be looking at something other than iptables – Peter Aug 29 '13 at 05:47
  • Maybe also try adding this `-A RH-Firewall-1-INPUT -p tcp --dport 80 -j LOG` – Lawrence Aug 29 '13 at 07:30
0

Try sudo service iptables stop to see if it is iptables-related.

Michel Samia
  • 181
  • 1
  • 3