There is the iptables quota module documented here: http://ipset.netfilter.org/iptables-extensions.man.html#lbBT
quota
Implements network quotas by decrementing a byte counter with each
packet. The condition matches until the byte counter reaches zero.
Behavior is reversed with negation (i.e. the condition does not match
until the byte counter reaches zero).
[!] --quota bytes
The quota in bytes.
I don't think this will help you though as, using this method, you would probably have to create individual quota rules for every possible source IP address that might come into your server (which is pretty much impossible). You would also need to integrate some kind of BASH script automation to reset these quotas every 24 hours (or what ever period), as mentioned here: IPtables Traffic Quota - up and down
Many years ago now, I toyed with an Apache module that was able to do this at the application level per IP address. This might be a better approach (I'm not sure what modern modules are available for Apache/Nginx that would do this - further research required).
The only other alternative that I know would work better (in my opinion) is using iptables hashlimit module in conjunction with tc (traffic shaping/policing): http://ipset.netfilter.org/iptables-extensions.man.html#lbAY - using this approach, traffic is only slowed but never blocked. The reason this might be more feasible is because the iptables hashlimit module can handle on-the-fly tracking of clients by recording distinct IP address and/or source/destination port "hash keys" as they come into your server. I have posted an answer on how to achieve this on another related question, here: https://serverfault.com/a/754116/30506 - but in summary, you create a traffic shaping class (bandwidth limiter pinned at say, 5Mb/sec) and then as the packet rate of per client sessions reaches a particular threshold, you start funneling their packets into this class, thus pacifying their traffic. In your case though, the pacification would need to be tuned to operate over hours of time rather than seconds.