5

I added a few thousand IPs packed in multiple /24 classes to a CentOS box and because of that I am getting scanned day and night. Since this server will be a workhorse and it will only need to be available on SSH (changed SSH port) and http/s, I ended up blocking mail ports and ping replies. That resolved the high response time over http/s, but iptables increased the load on the server.

I am getting:

top - 22:01:30 up 2 days, 10:17,  3 users,  load average: 1.99, 2.23, 2.07
Tasks: 198 total,   2 running, 195 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.0%us,  5.1%sy,  0.0%ni, 85.0%id,  0.0%wa,  0.0%hi,  8.8%si,  0.0%st
Mem:  16638532k total,  2692948k used, 13945584k free,   206992k buffers
Swap:  8193140k total,        0k used,  8193140k free,  2093204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7091 root      23   0 25704  23m  512 R 72.5  0.1   0:02.26 iptables
28507 root      15   0 10236 3308 2684 S  4.5  0.0   0:51.00 sshd

How can I resolve this issue, since the server is very slow when doing even basic stuff, like accepting a command via SSH ? I will be looking at a hardware firewall in the upcoming months, but until then the issue has to be resolved with a software solution.

petru
  • 113
  • 2
  • 6
  • 1
    You might want to share your iptables ruleset(s) with us so we can see if there's some room for improvements there. – Simon Strasser Jan 02 '13 at 20:22
  • What *problem* is the load average causing? Are you seeing increased latency in packets? Try using `tcpdump` (which is outside the firewall) and `strace` on a receiving service to determine the latency iptables is adding. – Ladadadada Jan 02 '13 at 20:31
  • 3
    That is showing the `iptables` command actually running. Are you running it constantly to get statistics or something? What command line are you passing it? (`ps axl` will tell you) – David Schwartz Jan 02 '13 at 20:33
  • @Simon, I am pasting just some of it, to pastebin. Your comment opened my eyes, I have over 40.000 lines of rules. Here is the pastebin link: http://pastebin.com/MjZsv1UE Can you help me improve my rules? Ladadadada well for a start, I see a large latency when typing into SSH. This latency is also observed when over iLO (HP KVM), so it is not network-side/firewall-side. Also the latency can be seen in any activity of the server. David, I am not trying to get stats at all. It looks like the load iptables makes is due to very many lines of rules, which obviously need to optimised. – petru Jan 02 '13 at 21:47
  • @petru: If you're not trying to get stats, why are you running the `iptables` command? What is its command line? – David Schwartz Jan 03 '13 at 04:11
  • God - 40.000 lines of rules? That smells like creep to me. I have a problem imagining a scenario where this makes sense. Your pastebin link does not show that though - there are a LOT less lines there. Anyhow. some ideas: * Use address list by service, * Use connection tracking and do not check permission for connections already established (i.e. check on new connections, once you say it is ok, it stays so - so the compelx logic whether something is allowed only happens once). – TomTom Jan 03 '13 at 06:46
  • In my mikrotik setup MOST packtets just get an "ok" because the connection has already been established, so all the further checks are not needed. I Only check on new connections and either ok or refuse them. – TomTom Jan 03 '13 at 06:47

2 Answers2

12

There are a few things you can do to optimise your iptables:

  1. Remove rules that don't do anything. Any rule without a target ( -j <something> ) doesn't do anything to the packet, so unless you are using these rules for accounting, they are useless and should be removed.

  2. Split rules into chains. A packet has to be matched against every rule up until it finds a target or the end of the chain where it uses the policy. If you create chains for groups of IPs that can be matched with a simple rule, you can drastically reduce the number of rules that have to be compared against for each packet. Something like this:

    :group0 ACCEPT [0:0]
:group1 ACCEPT [0:0]
:group2 ACCEPT [0:0]
:group3 ACCEPT [0:0]
:group4 ACCEPT [0:0]
-A INPUT -d 10.0.0.0/24 -j group0
-A INPUT -d 10.0.0.1/24 -j group1
-A INPUT -d 10.0.0.2/24 -j group2
-A INPUT -d 10.0.0.3/24 -j group3
-A INPUT -d 10.0.0.4/24 -j group4
-A group0 -d 10.0.0.1 -i ! lo -p tcp -m tcp --sport 80
...etc

    And then put all the rules for 10.0.0.0/24 in the group0 chain. Assuming you had one rule for each of the 65,536 IP addresses in 10.0.0.0/16, the worst-case number of comparisons a packet would have to go through in this set up is 512. Much better than the 65,536 comparisons every packet has to go through if they are all in the same chain. It doesn't have to be the destination IP range that you split on, it can be the port number or any other aspect of a packet that IPTables can match.

  3. If you are using this for accounting purposes, use a mirroring port on your switch to send a copy of all this traffic to a separate box. Have the primary routing firewall run with the minimum number of rules that achieves your security goals. (There may also be a better way of handling accounting other than iptables but that's not my area.)

  4. Put targets in for every rule. Unless you want to match all three of the following rules, you might as well have a -j ACCEPT on every rule. If you do want to match all of them, have the ACCEPT on the last one. That way you don't have to do comparisons for rules that will never match.

    -A acctboth -s 93.xyx.40.250 -i ! lo -p tcp -m tcp --dport 80
-A acctboth -s 93.xyx.40.250 -i ! lo -p tcp
-A acctboth -s 93.xyx.40.250 -i ! lo

  5. Consolidate rules. Since the logic in your firewall is very simple, you could replace the whole thing with about 10 rules - just the ones with -j TARGET.

  6. Remove duplicate and redundant rules. These rules don't look very well ordered or organised. I would be surprised if you didn't have duplicate rules in there somewhere. I would also be surprised if, with a few thousand IPs, some of the rules you have are for IPs you no longer have. Find these and remove them.

  7. Don't combine the chains for INPUT and OUTPUT. Put the -s <IP> rules in the OUTPUT chain and the -d <IP> rules in the INPUT chain and get rid of the -j acctboth rules and the acctboth chain. Again, this halves the number of rules that have to be checked before a match is found.

  8. Order the rules so that the most frequently matched targets are listed first. You can see human-readable counts with iptables -L -nv and raw numbers with iptables -L -nvx. If you have packets that benefit more from low latency (such as packets on port 22) put them earlier in the list than packets that benefit less (such as port 25).

Ladadadada
  • 25,847
  • 7
  • 57
  • 90
  • Very well said. This is not an art, but with some decent thinking you can really cut down on the number of rules that are to be processed. Likely by 90% for the lines, and more than 99% for the processing (by stopping fast). – TomTom Jan 03 '13 at 10:59
4

Add a hardware firewall. MIkrotik has terrific new ones - SERIOUSLY, 2 cores for every network port, running at 1.2ghz.

In the meantime you could try to optimize your iptable rules. THat pretty much is it - if rule processing takes so much time, optimize them to be faster.

TomTom
  • 50,857
  • 7
  • 52
  • 134
  • I agree, this will be the best solution in the end, but in the following months. Thank you for company tip, do you have any particular model in mind ? And do you have any other companies that produce such firewalls in mind ? – petru Jan 02 '13 at 22:18
  • Depends what you DO. Mikrotiks is quite unique in their price point - brutally cheap. Depending on how mmuch you must route there externally a 750 or 450G should be enough. I now use a 1100AH but it also switches my offices (it is in hte house center, from there we have 5 ports with a switch chip to server room and 4 offices). If you need a LOT of bandwidth, the CCR models can do millions of packets, but I leep that bandwidth out of the router and in my racks with their own routing switches. – TomTom Jan 02 '13 at 22:24