Answering my own question, after much work and research I created this script for iptables:
#!/bin/sh
/sbin/iptables -N BLOCK_IP
/sbin/iptables -N SYN_CHECK
/sbin/iptables -N DOS_CHECK
/sbin/iptables -N SYN_ATTACK
/sbin/iptables -N DOS_ATTACK
#
# first checks if the IP is already blocked
/sbin/iptables -A INPUT -j BLOCK_IP
# drop if is blocked
/sbin/iptables -A BLOCK_IP -p tcp -m multiport --dport 80,443 -m recent --name BlockedIP --rcheck --seconds 60 -j DROP
/sbin/iptables -A BLOCK_IP -p udp -m multiport --dport 80,443 -m recent --name BlockedIP --rcheck --seconds 60 -j DROP
# if already pass the time unblock the IP
/sbin/iptables -A BLOCK_IP -p tcp -m multiport --dport 80,443 -m recent --name BlockedIP --remove -j RETURN
/sbin/iptables -A BLOCK_IP -p udp -m multiport --dport 80,443 -m recent --name BlockedIP --remove -j RETURN
#
# check: if there is more than 20 simultaneous connections with SYN status - ignores IP Varnish Cache
/sbin/iptables -A INPUT -p tcp -m multiport --dport 80,443 --syn ! -s 123.456.789.000 -m connlimit --connlimit-above 20 -j SYN_CHECK
# check: hit and then connect frequency - ignores IP Varnish Cache
/sbin/iptables -A INPUT -p tcp -m multiport --dport 80,443 ! -s 123.456.789.000 -m state --state NEW -j DOS_CHECK
/sbin/iptables -A INPUT -p udp -m multiport --dport 80,443 ! -s 123.456.789.000 -m state --state NEW -j DOS_CHECK
#
# checks if the attack is frequently
/sbin/iptables -A SYN_CHECK -m recent --update --seconds 10 --hitcount 20 --name RATE -j SYN_ATTACK
/sbin/iptables -A DOS_CHECK -m recent --update --seconds 3 --hitcount 20 --name RATE -j DOS_ATTACK
# if the attack is frequent blocks for 1 minute and generates log
/sbin/iptables -A SYN_ATTACK -j LOG --log-prefix "BLOCK SYN ATTACK: " --log-level 6
/sbin/iptables -A SYN_ATTACK -m recent --set --name BlockedIP -j DROP
/sbin/iptables -A DOS_ATTACK -j LOG --log-prefix "BLOCK DOS ATTACK: " --log-level 6
/sbin/iptables -A DOS_ATTACK -m recent --set --name BlockedIP -j DROP
#
# if the attack is not frequent, accept
/sbin/iptables -A SYN_CHECK -m recent --set --name RATE -j ACCEPT
/sbin/iptables -A DOS_CHECK -m recent --set --name RATE -j ACCEPT
#
But I'm not sure if I'm totally sure.
In my opinion after seeing many examples on the internet this is one of the best protection scripts to http for iptables.
However, I would like another opinion, in my logic all makes sense. But I never programmed for iptables before.
I would like the opinion of an expert on that subject.