I have done this using a combination of TC and iptables hashlimit. I created a TC bandwidth limiter outbound on the LAN interface (to target download traffic) set to 5 Mbits/second. I then use the iptables hashlimit module in the interface's output mangle chain such that if the packet rate exceeds a certain threshold, between any two distinct source and destination IP addresses, it starts to classify those packets into the TC 5 Mbit/s traffic shaping class.
You have to set the packet threshold correctly, though, and I based mine on the fact that my MTU is 1500 bytes so used this to calculate how many packets per second that would be to create a 1.5 Mbits per second threshold (1000 packets/second). Coupled with setting the highest possible burst value on the hashlimit iptables module (which on my setup, appears to be 10,000), the result of all this is that short downloads run at full speed, but longer ones begin to slow down as some packets are passed into the TC bandwidth limiting class, which then obviously lowers the packet rate, etc etc. It is a bit of a hack, but amazingly it does work, and it works well. And because TC is used, no packets are ever dropped - simply delayed and slowed down due to being shoved into the TC class once the packet rate is reached.
This is on a CentOS 6 box. I think the more modern stuff allows for the hashlimit module to support bytes per second, not just packets per second, which is even better, but I tried this on my setup and it just reverts back to using packets per second.
I'm on a mobile phone just now so can't paste any config but if you want some examples, let me know and I'll edit this answer. I really like this solution because the limiting I've gone for is based on source and destination IP. So the system sees each distinct src+dst IP combination as its own packet stream for limiting. The hashlimit module supports source and destination ports as well (basically any combination of source IP, source port, destination IP, destination port), so you could even do the limiting on a per session basis between source IP+port and destination IP+port.
UPDATE
So below is roughly how you do it; experimentation will be required.
tc:
/sbin/tc qdisc add dev eth0 root handle 1: htb
/sbin/tc class add dev eth0 parent 1: classid 1:1 htb rate 5Mbit
iptables:
#!/bin/bash
[[ "$1" =~ ^[ADI]$ ]] || exit 1
for IPT in /sbin/ip{,6}tables
do
$IPT -t mangle -$1 POSTROUTING -o eth0 -m hashlimit --hashlimit-above 1000/second --hashlimit-burst 10000 --hashlimit-mode srcip,dstip --hashlimit-name limiter -j CLASSIFY --set-class 1:1
done