Condense iptables CIDR list?

Question

I have a list of about 170,000 IP ranges I would like to block ( Basically all IP's outside of the US and Canada ). The list is in CIDR format.

Is there any software that I can pipe this into and get the most condensed form of it?

I would like to input this in to iptables, but heard the practical limit for iptables is around 25,000

An example of the input file,

1.0.0.0/24
1.0.1.0/24
1.0.2.0/23
1.0.4.0/22
1.0.8.0/21
1.0.16.0/20
1.0.32.0/19
1.0.64.0/18
1.0.128.0/17
1.1.0.0/24

and just some manual conversion 1.0.1.0/24 and 1.0.0.0/24 could be summed up as 1.0.0.0/23 which when joined with 1.0.2.0/23 could be 1.0.0.0/22 which when joined with 1.0.4.0/22 would be 1.0.0.0/21 and so on

So manually condensing this would be

1.0.0.0/16
1.1.0.0/24

Answer 1

I found a tool that does it https://sourceforge.net/projects/cidrmerge/

cat unopt.txt | ./cidrmerge > optimized.txt

Answer 2

Look into ipsets. You can put these all into one rule that you can dynamically update without having to reload the rules. I have had really good success with using this to handle things like this. Here is a link to the man page for ipsets.

It is very simple to use once installed:

ipset create <nameofset> <typeofset>

In your case:

ipset create blockset hash:net

Then add subnets to the set:

ipset -A blockset 1.1.1.1/24

Then Add the rules to iptables to do what you want with the traffic.

iptables -A INPUT -m set --set blockset src -j DROP

Just an example. There are tons of things you can do with ipsets.