0

I'm hosting some relatively large files on my website, and it struck me that it would be trivial for someone to create a script that keeps downloading them over and over again, chewing through my data transfer and costing me money. Your typical home connection would be fast enough to cause me trouble if left downloading 24/7, nevermind a super fast fibre connection or a proper remote server.

So I'm wondering if there is anything available for Apache or even Ubuntu system-wide which will impose restrictions per IP address? For example, 10GB transfer per 24 hours. When the limit is reached, the client would be given a lightweight "forbidden - quota reached" page or be outright refused connection. I've looked around and haven't found anything other than various throttling solutions, which may help but wouldn't solve the issue.

spacer GIF
  • 101
  • 2
  • You may want to define your parameters better, there are lots of ways to skin a cat (Apache level? Iptables? QoS?). If you have all the files in a specified subdirectory and this is "all time/personal", I might set up fail2ban to block access for X hours if files in the appropriate directory are accessed from a given IP (determined by fail2ban parsing appropriate Apache log file) more then Y times in Z period. There are limits to the practicality of this approach in terms of not differentiating small and large files and handling NAT – davidgo Jan 15 '20 at 07:16
  • Let me know if you want me to see if I can provide a ruleset for fail2ban - it occurs to me I've done something similar to limit WordPress login bruteforce abuse on some of my servers. – davidgo Jan 15 '20 at 07:17
  • A MUCH more complex variant might be to tie in f2b and iptables quotas. Not played with this though. An easy mechanism might be a fallback limiting/blocking all traffic coming in to port 80 after a quota has been met. – davidgo Jan 15 '20 at 07:22
  • @davidgo I have full control over the server so I'm open to whatever might work best. Recording the number of accesses wouldn't be an ideal solution because the server could get thousands of hits during a browsing session with small images galore (and I certainly wouldn't want to block Googlebot etc.) but download large files thousands of times and that's a lotta data. – spacer GIF Jan 16 '20 at 02:29
  • Couple of things - The idea was to have the large files in a seperate subdirectory, so that only hits against those files are recorded. (presumably you would use a robots.txt file to stop Google searching that directory as well). Are you in a position to make the files web accessible only through a customised web page? If so, you could log the download request and IP address in a database, and deny once your criteria have been reached? – davidgo Jan 16 '20 at 02:36
  • @davidgo I could seperate large files from the rest. My concern about logging requests would be how partial downloads are handled. For things like ZIPs and high-resolution images this should be fine, but if I have a media file someone keeps seeking about in it's not much use if that gets logged as a big data drain and the user gets blocked without actually having any malicious intention. – spacer GIF Jan 16 '20 at 03:00

2 Answers2

0

There is the iptables quota module documented here: http://ipset.netfilter.org/iptables-extensions.man.html#lbBT

quota
    Implements network quotas by decrementing a byte counter with each
    packet. The condition matches until the byte counter reaches zero.
    Behavior is reversed with negation (i.e. the condition does not match
    until the byte counter reaches zero).

    [!] --quota bytes
           The quota in bytes.

I don't think this will help you though as, using this method, you would probably have to create individual quota rules for every possible source IP address that might come into your server (which is pretty much impossible). You would also need to integrate some kind of BASH script automation to reset these quotas every 24 hours (or what ever period), as mentioned here: IPtables Traffic Quota - up and down

Many years ago now, I toyed with an Apache module that was able to do this at the application level per IP address. This might be a better approach (I'm not sure what modern modules are available for Apache/Nginx that would do this - further research required).

The only other alternative that I know would work better (in my opinion) is using iptables hashlimit module in conjunction with tc (traffic shaping/policing): http://ipset.netfilter.org/iptables-extensions.man.html#lbAY - using this approach, traffic is only slowed but never blocked. The reason this might be more feasible is because the iptables hashlimit module can handle on-the-fly tracking of clients by recording distinct IP address and/or source/destination port "hash keys" as they come into your server. I have posted an answer on how to achieve this on another related question, here: https://serverfault.com/a/754116/30506 - but in summary, you create a traffic shaping class (bandwidth limiter pinned at say, 5Mb/sec) and then as the packet rate of per client sessions reaches a particular threshold, you start funneling their packets into this class, thus pacifying their traffic. In your case though, the pacification would need to be tuned to operate over hours of time rather than seconds.

parkamark
  • 1,118
  • 6
  • 11
  • While this isn't a definitive solution, I can play with `hashlimit` and see if I can get it to behave roughly how I want. Cheers for the links! – spacer GIF Jan 16 '20 at 02:41
-2

Hello and welcome to Server Fault!

Bandwidth limiting can be done using iptables and the limit module.

A very good introduction to this can be found here

A small example from this link to set bandwidth throtteling up:

$ sudo iptables --flush  # start again
$ sudo iptables --new-chain RATE-LIMIT
$ sudo iptables --append INPUT --match conntrack --ctstate NEW --jump RATE-LIMIT
$ sudo iptables --append RATE-LIMIT --match limit --limit 50/sec --limit-burst 20 --jump ACCEPT
$ sudo iptables --append RATE-LIMIT --jump DROP

Kudos to the authors Will Sewell and Jim Fisher of the mentioned article!

Reiner Rottmann
  • 623
  • 1
  • 7
  • 19
  • 1
    Not what the OP asked for - "For example, 10GB transfer per 24 hours. When the limit is reached, the client would be given a lightweight "forbidden - quota reached" page or be outright refused connection." - the idea is to limit amount, NOT to limit transfer speed which would be totally crappy (ALL downloads get slower, even the first ones). – TomTom Jan 15 '20 at 08:22
  • Bandwith is throtteled indirectly. A TCP packet is maximum 65535 bytes. `--limit 10737418240/day` would limit to 10GB a day. `--cstate NEW,ESTABLISHED` would be needed in order to limit packages from existing connections. – Reiner Rottmann Jan 15 '20 at 12:22
  • 1
    Yes, but the problem is that I would want to limit you brutally after 10g - but the first 10g please come as fast as possible ;) – TomTom Jan 15 '20 at 20:21
  • If you download below 10gb/day, the speed per ip will be total bandwith of server divided by the number of simultanous downloads. So as fast as possible. A jump to drop will brutaly limit you. A jump to reject will give the client the courtesy that we rejected his packages. After each day, the counters will be reset. – Reiner Rottmann Jan 16 '20 at 08:20