iptables blocking all traffic even with ports opened

Question

Simple as I can put it: I can't access my servers with iptables enabled, with default rules. Need to disable iptables to get access. Trying to find the cause. Only difference in setup type is layer3 switch instead of layer2, but have not configured the layer3 switch in any way other then enabling IP routing (no tagging/untagging has been done on any VLAN). If I understood why the packets were being blocked, I'd definitely try to clear up my question! ;)

More info on the matter:

Having an issue with a new minimal CentOS 6.5 install and iptables. This is a new datacenter location I'm trying to prepare to bring some servers online, but can't just yet. iptables seems to be dropping legitimate packets, unless a specific rule is in place to allow that certain IP through (not ideal for web servers). We use OpenVZ and install cPanel within the VPS node, as well as csf firewall. All settings are configured correctly on the node, same as my other working servers in other datacenters, so its not an iptables module issue.

Having setup plenty of other servers without issue in the past, in other DC's, the only difference here is (of course) the different ISP, and also the fact that I'm using a Layer3 switch (Baystack 5510), instead of a Layer 2 switch. I'm not sure if this is the cause of the issue or not, testing directly connected to the ISP connection tomorrow when I have access.

Could this have to do with improper setup of my layer3 switch? Basically I have the ISP's ethernet cable connected to port 48, and my server connected to port 1 & 2 (bond0), tried using just default eth0 too. Default configs, only difference is I have IP routing enabled so that I could add other IP subnets in later. I did not set ANY tagging/untagging, which I'm now reading up on a bit, but could this be what is causing iptables to flag/drop these packets?

This is a basic install that seems to be blocking packets with default CentOS iptables rules, or at least when I switched to OpenVZ kernel, had to drive to DC to get physical access and do 'service iptables stop' and 'chkconfig iptables off' to prevent from losing further access.

Basically the ports are opened in iptables, but I still cannot get access to the servers with iptables running. Tried this on 2 severs, fresh and minimal CentOS 6.5 install. I ran a Wireshark & tcpdump to capture network traces on both ends during this process. This is the first packet, attemping a [SYN] packet to initiate a TCP session on port 80, which is opened in iptables. This packet does not get a reply ([SYN ACK]), but instead goes through TCP Retransmission over and over, but it IS received on the server end, but dropped with either TCP_IN Blocked or UDP or OUT, or invalid packet:

Client side (my PC): 77 4.434317000 192.168.2.244 SERVER-IP TCP 66 63866 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=4 SACK_PERM=1

Server side (server tcpdump) 1 0.000000 MY-ISP-IP SERVER-IP TCP 68 61992 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1452 WS=4 SACK_PERM=1

Here's the invalid packet: kernel: [16708.550424] Firewall: INVALID IN=venet0 OUT= MAC= SRC=MYIP DST=SERVERIP LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=12281 DF PROTO=TCP SPT=60992 DPT=80 WINDOW=8192 RES=0x00 SYN URGP=0

Here's a blocked port 21 (opened also) packet: kernel: [20604.837769] Firewall: TCP_IN Blocked IN=venet0 OUT= MAC= SRC=MYIP DST=SERVERIP LEN=48 TOS=0x00 PREC=0x00 TTL=112 ID=15851 DF PROTO=TCP SPT=61742 DPT=21 WINDOW=8192 RES=0x00 SYN URGP=0

Answer 1

This looks like a problem in your OpenVZ configuration, mainly related with the state feature of netfilter.

Make sure that in your /etc/vz/vz.conf, IPTABLES_MODULE contains ipt_state. This is important for older versions of vzctl, but better be safe than sorry:

## IPv4 iptables kernel modules to be enabled in CTs by default
IPTABLES="ipt_REJECT ipt_tos ipt_limit ipt_multiport iptable_filter iptable_mangle ipt_TCPMSS ipt_tcpmss ipt_ttl ipt_length ipt_state"
## IPv4 iptables kernel modules to be loaded by init.d/vz script
IPTABLES_MODULES="$IPTABLES"

Also, in each of your /etc/vz/conf/*.conf, if NETFILTER is supported, it should be set to stateful:

NETFILTER="stateful"

This new NETFILTER configuration parameter has been added lately. This might be why you have not heard of it anywhere else. You can see the commit here.