Reloading IPTables Disable Access to Site

Question

I have CentOS 7 VPS.

IPtables contains 2000 IP block rules. When reloading it takes 80 seconds to complete reloading.

During that 80 seconds, I notice that around 15 seconds, my site becomes unresponsive and no page can be visited from browser. When reload completes, site comes back and pages can be visited.

I want to ask is network unresponsiveness expected when IPTables rules are reloaded again?

This is main structure:

/sbin/iptables -F FORWARD
/sbin/iptables -F OUTPUT
/sbin/iptables -Z FORWARD
/sbin/iptables -Z OUTPUT

/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -p tcp ! --syn  -m state --state NEW -j REJECT --reject-with tcp-reset
/sbin/iptables -A INPUT -m state --state INVALID -j DROP
/sbin/iptables -P OUTPUT DROP
/sbin/iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -p tcp ! --syn  -m state --state NEW -j REJECT --reject-with tcp-reset
/sbin/iptables -A OUTPUT -m state --state INVALID -j DROP
/sbin/iptables -P FORWARD DROP
/sbin/iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A FORWARD -p tcp ! --syn  -m state --state NEW -j REJECT --reject-with tcp-reset
/sbin/iptables -A FORWARD -m state --state INVALID -j DROP

/sbin/iptables -A INPUT -i lo  -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
/sbin/iptables -A FORWARD -i lo -o lo -j ACCEPT

/sbin/iptables -t mangle -F
/sbin/iptables -t mangle -Z
/sbin/iptables -t mangle -P PREROUTING ACCEPT
/sbin/iptables -t mangle -P OUTPUT ACCEPT
/sbin/iptables -t mangle -P INPUT ACCEPT
/sbin/iptables -t mangle -P FORWARD ACCEPT
/sbin/iptables -t mangle -P POSTROUTING ACCEPT

/sbin/iptables -t nat -F
/sbin/iptables -t nat -Z
/sbin/iptables -t nat -P PREROUTING ACCEPT
/sbin/iptables -t nat -P OUTPUT ACCEPT
/sbin/iptables -t nat -P POSTROUTING ACCEPT

#
# ACCEPT RULES. JUST A FEW
#
#
/sbin/iptables -I INPUT -s 95.104.79.33 -j ACCEPT
/sbin/iptables -I INPUT -s 108.161.176.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT

#
# BLOCK RULES. EXCERPT. 2000 RULES MORE
#
#
/sbin/iptables -A INPUT -s 9.101.144.4 -j DROP
/sbin/iptables -A INPUT -s 94.229.64.6 -j DROP
/sbin/iptables -A INPUT -s 973.209.49.7 -j DROP
/sbin/iptables -A INPUT -s 98.150.131.8 -j DROP
/sbin/iptables -A INPUT -s 98.168.114.5 -j DROP

/sbin/iptables -A INPUT -p tcp --dport 12443 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 11443 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 11444 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 8447 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 8443 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 8880 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 80 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 443 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 21 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 587 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 25 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 465 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 110 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 995 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 143 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 993 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 106 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 3306 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 5432 -j ACCEPT

/sbin/iptables -A INPUT -p tcp --dport 9008 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 9080 -j ACCEPT

/sbin/iptables -A INPUT -p udp --dport 137 -j ACCEPT
/sbin/iptables -A INPUT -p udp --dport 138 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 139 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 445 -j ACCEPT

/sbin/iptables -A INPUT -p udp --dport 1194 -j ACCEPT

/sbin/iptables -A INPUT -p udp --dport 53 -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 53 -j ACCEPT

/sbin/iptables -A INPUT -p icmp --icmp-type 8/0 -j ACCEPT

/sbin/iptables -A INPUT -j DROP

/sbin/iptables -A OUTPUT -j ACCEPT

/sbin/iptables -A FORWARD -j DROP

Actually I am using Plesk firewall reload shell script: firewall-active.sh

firewall-active.sh

#!/bin/bash
#ATTENTION!
#
#DO NOT MODIFY THIS FILE BECAUSE IT WAS GENERATED AUTOMATICALLY,
#SO ALL YOUR CHANGES WILL BE LOST THE NEXT TIME THE FILE IS GENERATED.

set -e

echo 0 > /proc/sys/net/ipv4/ip_forward
([ -f /var/lock/subsys/ipchains ] && /etc/init.d/ipchains stop) >/dev/null 2>&1 || true
(rmmod ipchains) >/dev/null 2>&1 || true

apply_rule()
{
    iptables_bin="$1"
    shift

    iptables_version=`/sbin/iptables --version | awk -F '.' '{print $2$3}'`

    # Use the native --wait option since v1.4.20
    if [ $iptables_version -gt 420 ]; then
        $iptables_bin -w $@ 2>/dev/null
        return $?
    fi

    # Emulate --wait for elderly versions
    for i in `seq 10`; do
        $iptables_bin $@ 2>&1 | grep -q xtable || return 0
        sleep 1
    done

    return 1
}

#
# CREATE DELETE RULES BY USING IPTABLES-SAVE
#
#
/sbin/iptables-save  -t filter | grep -- "-A INPUT" |  grep -v "fail2ban-\|f2b-" | sed -e "s#^-A#apply_rule /sbin/iptables -D#g" | (echo -e "`declare -f apply_rule`\n" && cat ) | /bin/bash

apply_rule /sbin/iptables -F FORWARD
apply_rule /sbin/iptables -F OUTPUT
apply_rule /sbin/iptables -Z FORWARD
apply_rule /sbin/iptables -Z OUTPUT

apply_rule /sbin/iptables -P INPUT DROP
apply_rule /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp ! --syn  -m state --state NEW -j REJECT --reject-with tcp-reset
apply_rule /sbin/iptables -A INPUT -m state --state INVALID -j DROP
apply_rule /sbin/iptables -P OUTPUT DROP
apply_rule /sbin/iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
apply_rule /sbin/iptables -A OUTPUT -p tcp ! --syn  -m state --state NEW -j REJECT --reject-with tcp-reset
apply_rule /sbin/iptables -A OUTPUT -m state --state INVALID -j DROP
apply_rule /sbin/iptables -P FORWARD DROP
apply_rule /sbin/iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
apply_rule /sbin/iptables -A FORWARD -p tcp ! --syn  -m state --state NEW -j REJECT --reject-with tcp-reset
apply_rule /sbin/iptables -A FORWARD -m state --state INVALID -j DROP

apply_rule /sbin/iptables -A INPUT -i lo  -j ACCEPT
apply_rule /sbin/iptables -A OUTPUT -o lo -j ACCEPT
apply_rule /sbin/iptables -A FORWARD -i lo -o lo -j ACCEPT

apply_rule /sbin/iptables -t mangle -F
apply_rule /sbin/iptables -t mangle -Z
apply_rule /sbin/iptables -t mangle -P PREROUTING ACCEPT
apply_rule /sbin/iptables -t mangle -P OUTPUT ACCEPT
apply_rule /sbin/iptables -t mangle -P INPUT ACCEPT
apply_rule /sbin/iptables -t mangle -P FORWARD ACCEPT
apply_rule /sbin/iptables -t mangle -P POSTROUTING ACCEPT

apply_rule /sbin/iptables -t nat -F
apply_rule /sbin/iptables -t nat -Z
apply_rule /sbin/iptables -t nat -P PREROUTING ACCEPT
apply_rule /sbin/iptables -t nat -P OUTPUT ACCEPT
apply_rule /sbin/iptables -t nat -P POSTROUTING ACCEPT

#
# ACCEPT RULES. JUST A FEW
#
#
apply_rule /sbin/iptables -I INPUT -s 95.104.79.33 -j ACCEPT
apply_rule /sbin/iptables -I INPUT -s 108.161.176.0/20 -p tcp -m multiport --dports 80,443 -j ACCEPT

#
# BLOCK RULES. EXCERPT. 2000 RULES MORE
#
#
apply_rule /sbin/iptables -A INPUT -s 9.101.144.4 -j DROP
apply_rule /sbin/iptables -A INPUT -s 94.229.64.6 -j DROP
apply_rule /sbin/iptables -A INPUT -s 973.209.49.7 -j DROP
apply_rule /sbin/iptables -A INPUT -s 98.150.131.8 -j DROP
apply_rule /sbin/iptables -A INPUT -s 98.168.114.5 -j DROP

apply_rule /sbin/iptables -A INPUT -p tcp --dport 12443 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 11443 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 11444 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 8447 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 8443 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 8880 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 80 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 443 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 21 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 22 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 587 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 25 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 465 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 110 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 995 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 143 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 993 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 106 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 3306 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 5432 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p tcp --dport 9008 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 9080 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p udp --dport 137 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p udp --dport 138 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 139 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 445 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p udp --dport 1194 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p udp --dport 53 -j ACCEPT
apply_rule /sbin/iptables -A INPUT -p tcp --dport 53 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -p icmp --icmp-type 8/0 -j ACCEPT

apply_rule /sbin/iptables -A INPUT -j DROP

apply_rule /sbin/iptables -A OUTPUT -j ACCEPT

apply_rule /sbin/iptables -A FORWARD -j DROP

echo 1 > /proc/sys/net/ipv4/ip_forward
#
# End of script
#

What it does actually is to:

first check iptables-save,
find -A INPUT rules,
convert them to -D rules to delete them.

then reload firewall as seen above structure. Because there are 2000 separate rules, it takes 80 seconds and 15 seconds of which site is unresponsive.

I tried to combine those 2000 separate rule into 1 rule by specifying IPs side by side but it did not make sense. Because iptables added 2000 rules into the tables anyway. Reload decreased to 40 seconds but it is still slow.

Which part of the script is creating unresponsive please?

I've told you clearly in my answer which part of the script is making your server "*unresponsive*" - it's the policy combined with flushing the rules, and there's no real way around that. `iptables` isn't intended for what you're doing. — MadHatter, Jun 15 '16 at 19:22
Thank you very much for your help. I see. I can not use IPset because virtualization environment does not allow. I am thinking to block malicious users by route blackhole. Do you have any recommendation for blocking IPs please? — NecNecco, Jun 15 '16 at 21:42
For blocking IPs as you are trying to? Don't bother, or if there's something so odd about your site that noone from China must even try to load the front page, have Apache do it. Moreover, it sounds like you're not even using a VPS, but a mere container (hence inability to use ipset) - if performance is this critical to you, consider more decent hosting. *However* you do your geolocation, it's going to be expensive, and at least a proper VPS would give you a full range of tools. — MadHatter, Jun 16 '16 at 05:17

MadHatter · Accepted Answer · 2016-06-15T14:37:48.927

Firstly, yes, the unresponsiveness is expected; the reason is the reload starts off by setting policy and flushing the ruleset. The first thing your script does is flush the INPUT chain (actually, it does no such thing, but I suspect you've just left the first two lines off) then set policy to DROP. Once the policy is set and the existing ruleset is flushed, you've an empty chain with a DROP policy, which promptly lets no traffic pass. As the rules are subsequently reloaded, normal behaviour is resumed.

The bigger problem is the size of your ruleset. 2000 rules is, with respect, insane. I've managed some very big iptables-based firewalls - for whole sites, not just single machines - over the years, and never reached 500 rules, even after a decade in service. 2000 rules smacks to me very much of many years of "oooh dev just need to be able to get from this address to this address on this port", with no pushback, consolidation, nor any garbage collection / cleanup on the ruleset. A firewall with 2000 exceptions to a DENY policy is also, frankly, not so much a firewall as a poor-quality patch cable.

My strong feeling is that you need to call on your user community to justify every single rule in that ruleset, with a clear date on which any rule not still in use and business-necessary is going. You will likely find that some of the rules which need to be kept can be consolidated, with the use of user-defined chains, -m multiport, use of CIDR suffixes with -s and/or -d, and so on.

Until you get that ruleset heavily pruned, reloading it is going to take a while, and the firewall will not behave normally while you're doing it.

Thank you for your help. Most of the IPs come from ASIA which is not my site is targeted for. Yes I am a bit lazy to prune the IPtables rules but I did not have chance to prune it. I think, if I can not fix unresponsive issues, I am going to quit using iptables to block users but switch to **route blackholes**. — NecNecco, Jun 15 '16 at 14:09

Reloading IPTables Disable Access to Site

1 Answers1