How can I detect long-lived connections and mark them for shaping

Question

I would like to detect TCP connections which have been open for more than one minute (or for greater than N bytes, or M packets), so I can classify them as bulk traffic ("downloads") and de-prioritise them.

Can I detect long-running streams with iptables/netfilter/conntrack, and mark them for shaping by tc?

I thought I may be able to use the TCP "sequence number" as some measure of the length of a stream, but unfortunately it does not start at zero! Perhaps connection tracking with netfilter/conntrack could determine the correct sequence number, or the total number of bytes, and the duration of the connection, and choose whether to mark the packet.

(I could also mention that I am doing this on ingress using an ifb0 virtual interface. I am currently using tbf queues (with sfq) to limit low-priority data. Regardless, any solution could also be applied to egress, for example for a webserver on a limited connection who wants to de-prioritise downloads in order to speed up small requests.)

---

Update: Using conntrackd I am able to see a list of connections, and how long ago each one was initiated. I have started using this list to detect IP/port pairs which I want to limit (after 60 seconds). However there are number of issues:

• conntrack(d) does not appear to clear connections out of the list promptly when they have closed! So I end up marking all the connections for limiting, even ones which have finished.
• Setting marks in conntrack does not seem to set marks in the packets (as far as tc can see). (This is not just because conntrack sees the packets after ifb0: I cannot see any marks on egress packets either.) So I have just been adding new filters to tc for limiting, which is far from ideal in the long term.
• conntrack(d) doesn't tell me how much bandwidth each connection has used. So I cannot distinguish intermittent (e.g. iosocket) sessions from heavy downloads. (In any case, this is a difficult problem: if we have 5 downloads already and a new download starts, how can we tell that he is trying to flood? He won't reach anywhere near the max rate.)

Any suggestions with the above would be appreciated.

(On the bright side, even if I cannot classify the downloads correctly, simply using tfb to limit incoming data to 80% of my maximum downrate has prevented overflooding the connection, and has allowed new connections to establish much more easily than before.)

Answer 1

The life-expectancy and actual longevity of a connection has ZERO relation on whether a connection should be shaped specially or not.

Some examples:

SSH, possibly long-lived, minutes, hours, days, weeks, even months. Still needing high priority to be responsive to user interaction.

Bittorrent (or similar protocols), randomly lived, some for seconds then disconnected, others for minutes or hours, then disconnected. Few lasting more than a day at a time.

Summary: The length of a connection has NOTHING to do with whether the connection is "bulk" or not, and whether or not the connection should be treated as bulk.

Not necessarily directly related, but useful:

Using traffic shaping, is limiting download traffic helpful or harmful?