how to throttle http requests on a linux machine?

Question

EDIT: here is the summery: i need to reduce max connections preferably system wide on Ubuntu 11.04 but at least within Google Chrome. i do not need or want to throttle bandwidth, Verizon seems to only care about the number of connections so that is all i want to change. also, i don't want to use firefox unless i have to, i have three other machines all using chrome and synced and i just prefer it over firefox.

i use tethering for my home internet connection via my verizon cell phone. without paying for it. this works just fine for streaming netflix via my nintendo wii and pretty much every other conceivable use ive had for it. except, during heavy usage with multiple tabs open on my laptop, the network connection on my phone will just turn off, then on again, then off, but it never fully connects.

i think, based on this and other questions that this is caused by verizon getting too many http requests from my phone. is there some software, script, setting or otherwise that would allow me to throttle my requests to say, 5 or 10 or whatever it turns out is 1 less than verizon is looking for, so that my cell's network connection is not lost?

i would far prefer a slow down rather than complete shut off of my internet connection. i am almost certain is from quantity of requests and not related to data, because, as i mentioned, netflix will run all day without a hitch, and that uses more data than anything else i would be doing. if i had a router i am pretty sure there are settings i could easily change to only allow so many requests at a time ... but in this case, my phone is my router, so no settings. im using ubuntu 11.04 on my netbook with an htc incredible on verizon (not that the phone details are relevant)

i have been trying to figure this out for quite some time, currently the only fix is ensure that all requests are stopped and then sometimes it works again, other times i have to manually turn my 3g service off and then back on. thank you so much for any assistance!

Answer 1

I successfully use an SSH tunnel to circumvent arbitrary connection limits when tethering. The idea is as follows:

ssh -D 1234 server

starts the tunnel, then you configure firefox or other software to use socks 5 proxy localhost/1234. Follow the instructions on this page to instruct firefox to also use the proxy for DNS.

The result is that the router will only see one encrypted connection to your proxy.

If you are using Ubuntu, then you can configure the proxy settings for Ubuntu, not for Chrome or Firefox. Then, all programs including Chrome and Firefox should use that proxy by default.

Open gnome-network-properties, and set the proxy to host localhost, port 1234.

Answer 2

If you're using Firefox, search for "max-connections" settings in about:config.

Anything more complex than that likely requires traffic control / packet filtering / firewall -- for example, you can use Shorewall to do connection rate limiting: http://www.shorewall.net/ConnectionRate.html

Answer 3

You can add a rule to the netfilter/iptables firewall to do this for you. Netfilter consults the nat table concerning new connections (and the filter table per packet). You can rate-limit new connections using a single iptables command.

# iptables --table nat --append --protocol tcp -m multiport \
    --destination-ports 80,443 -m limit ! --limit 10/second --limit-burst 5 \
    --jump DROP

This command must be run as root. It matches all new --table-nat, tcp --protocol tcp connection attempts to ports 80 (http) and 443 (https) --destination-ports 80,443 which is above ! the limit of 5 connection attempts per second --limit 10/second, with an allowance of 5 extra connections --limit-burst 5. It then instructs netfilter to completely ignore these packets --jump DROP.

NB, you'll need to run this command every time you reboot your computer, which can be accomplished via a variety of mechanisms. You can throw this into a shell script, make it executable, and call it via a pre-up line in /etc/network/interfaces--if you use this file to configure your network, and if you use a debian variant.

An quick, easy hack would be to add this line to one of the important-looking start-up scripts under /etc/init.d/.

Answer 4

Per the Chromium thread Issue 12066: Match Firefox's per-host connection limit of 15, the Chrome team believes 6 connections to a server is the correct upper bound.

To change the number of connections above or below 6, will require building Chrome from source. Or passing to Firefox.

Here are the default values in Firefox (3.5), which can be changed in about:config :

network.http.max-connections 30
network.http.max-connections-per-server 15
network.http.max-persistent-connections-per-proxy 8
network.http.max-persistent-connections-per-server 6
network.http.pipelining false
network.http.pipelining.maxrequests 4
network.http.proxy.pipelining false

Documentation per each of the above settings can be found in the MozillaZine Knowledge Base.

Answer 5

For a low level tool or command line tool try tc (traffic control). This tool is built to work with the networking stack of most new kernels. So it should be available on most systems by default, no install necessary.

For http and tc http://www.cyberciti.biz/faq/linux-traffic-shaping-using-tc-to-control-http-traffic/

tc
man tc

http://wiki.openvz.org/Traffic_shaping_with_tc
http://www.topwebhosts.org/tools/traffic-control.php

Also worth reading http://www.faqs.org/docs/Linux-HOWTO/Bandwidth-Limiting-HOWTO.html

Answer 6

Trickle is a util that will do this for you.

Click here for more info

Trickle packages are available for most Linux distributions and BSD.

Here's a link to info for Trickle on the latest version of Ubuntu, with download links

It's probably available through either the Ubuntu Software Center or Synaptic Package Manager also.

Here's a link for Fedora

Answer 7

I suspect the best way to do this system-wide, is to use an HTTP proxy. You could then optionally configure it as a Transparent Proxy to force all connections to use it, but it's easier (and has fewer limitations) to just configure your web browsers manually.

The main reason to do this with a proxy, rather than with iptables, is so that it doesn't anger your web browser. By using the -j DROP, as suggested in another answer, you're essentially discarding the TCP connection entirely--which means your browser will probably sit there waiting for a response, until it times out, then will finally show you a half-loaded page, with broken windows, missing CSS, etc.

By using a proxy instead, the pending requests aren't dropped--they're just queued until there's room to continue.

Probably the simplest, lightest weight configuration (that I've found so far) would be to install Tinyproxy, and set the MaxClients value to the maximum number of outbound connections you want.

If you want to use squid--perhaps because you just like overkill, or maybe you like its caching or other fancy whiz-bang features, the only outbound connection limit I see is on a per-peer basis. But this can still work. Set up two proxies--one as squid, one as something else (perhaps Tinyproxy, hehe), and configure Tinyproxy as squid's only peer with the max-conn setting set to your maximum number of connections. Then tell squid to forward all requests through that peer by setting the never_direct option.