you can use modsecurity to define any rule you want, block intensive requests with time-based blocking, by counting requests, blacklist, run external app (i.e. block with firewall), forward to proxy, drop packet, reject connection, reset connection, redirect, display 404 error (ir maybe you like error 666), play with variables etc. It can do many things: https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual
Example: http://halfelf.org/2013/wp-login-protection-modsec/
SecAction phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},initcol:user=%{REMOTE_ADDR},id:5000134
<Locationmatch "/wp-login.php">
# deny status and log, 401 to client
SecRule user:bf_block "@gt 0" "deny,status:401,log,id:5000135,msg:'ip address blocked for 5 minutes, more than 10 login attempts in 3 minutes.'"
# Tracking: On a successful login, a 302 redirect is performed, a 200 indicates login failed.
SecRule RESPONSE_STATUS "^302" "phase:5,t:none,nolog,pass,setvar:ip.bf_counter=0,id:5000136"
# count last 3 minutes
SecRule RESPONSE_STATUS "^200" "phase:5,chain,t:none,nolog,pass,setvar:ip.bf_counter=+1,deprecatevar:ip.bf_counter=1/180,id:5000137"
# if matched 10 times, block for 5 minutes
SecRule ip:bf_counter "@gt 10" "t:none,setvar:user.bf_block=1,expirevar:user.bf_block=300,setvar:ip.bf_counter=0"
</locationmatch>
Also, you can use fail2ban to analyze logs and block client according to that.