9

I have a linux router which has a 10GBe interface on the outside and bonded Gigabit ethernet interfaces on the inside.

We have currently budget for 2GBit/s. If we exceed that rate by more than 5% average for a month then we'll be charged for the whole 10Gbit/s capacity. Quite a step up in dollar terms.

So, I want to limit this to 2GBit/s on 10GBe interface.

TBF filter might be ideal, but this comment is of concern.

On all platforms except for Alpha, it is able to shape up to 1mbit/s of normal traffic with ideal minimal burstiness, sending out data exactly at the configured rates.

Should I be using TBF or some other filter to apply this rate to the interface and how would I do it. I don't understand the example given here: Traffic Control HOWTO

In particular "Example 9. Creating a 256kbit/s TBF"

tc qdisc add dev eth0 handle 1:0 root dsmark indices 1 default_index 0
tc qdisc add dev eth0 handle 2:0 parent 1:0 tbf burst 20480 limit 20480 mtu 1514 rate 32000bps

How is the 256K bit/s rate calculated? In this example, 32000bps = 32k bytes per second. Since tc uses bps = bytes per second. I guess burst and limit come into play but how would you go about choosing sensible numbers to reach the desired rate?

This is not a mistake. I tested this and it gave a rate close to 256K but not exactly that.

EDIT

After doing a lot of reading and testing, I've come to the conclusion that TBF is inappropriate because of the bandwidth involved. Whatever settings I tried I couldn't get TBF to provide bandwidth > ~50Mbit/s. According to lartc.org/lartc.pdf, the RED method is better for shaping bandwidth > 100Mbit/s so I will attempt to use that.

However, choosing the a value for min (i.e. Average queue size at which marking becomes a possibility). The example given is this:

You should set the min by calculating that highest acceptable base queueing latency you wish, and multiply it by your bandwidth. For instance, on my 64kbit/s ISDN link, I might want a base queueing latency of 200ms so I set min to 1600 bytes.

  1. how would you choose the highest acceptable base queuing latency? The example is for 64kbit/s.

  2. What would be acceptable for 2Gbit/s?

hookenz
  • 14,132
  • 22
  • 86
  • 142
  • Oh this has come up again. I did find a way to limit the rate to 2Gbit/s successfully, so it's possible. I just can't recall how I did it. I think I found a script that got me most of the way. – hookenz Jul 19 '22 at 01:11

3 Answers3

2

  1. You should chose acceptable queueing latency basing on type of traffic.

    • For example, for voice queuing more that 200ms is already a problem.
    • While having 500ms buffer for ftp/torrent traffic is not a big problem at all.

  2. Queuing latency/strategy is a question of traffic type not interface speed. For example VOIP, perhaps, should not be queued at all. Unfortunately the tc RED documentation is not very clear, you would better read some RED info on Juniper/Cisco site and apply that knowledge to tc.

guntbert
  • 553
  • 7
  • 21
Alex_www
  • 136
  • 3
1

How is the 256K bit/s rate calculated? In this example, 32,000bps = [32,000] bytes per second.

Yes, the math there is correct. If you're seeing a number close to 256k, it's probably slightly below. Where are you measuring that number from? If it's your browser's downlaod or something similar, they don't count the overhead of packet headers, but tc counts everything.

Jeff Ferland
  • 20,239
  • 2
  • 61
  • 85
1

In my experience, qdisc TBF is easily able to limit bandwidth to 1 Gbps so I would guess it will also scale to 2 Gbps. However, you'll probably need a real CPU for the job instead of some low-end edge router. Something like 4 GHz i3 will surely be enough.

Try something like

tc qdisc add dev "$DEV" root handle 1: \
  tbf rate "$UPLINK_RATE" burst "$UPLINK_BURST" latency "$TBF_LATENCY"

where

DEV="$(ip route | grep "^default " | grep -Po "(?<=dev )[^ ]+")"
UPLINK_RATE="2000Mbit"
UPLINK_BURST="6500"
TBF_LATENCY="14ms"

Note that to use low latency TBF you may need to be running PREEMPT kernel (e.g. Ubuntu linux-lowlatency-hwe-* package) or the system may fail handling all those packages.

See also: https://networkengineering.stackexchange.com/a/54404/36597

Mikko Rantalainen
  • 858
  • 12
  • 27
  • Thanks. I forgot about this question. I did find an answer and had a script that did it. But I've left the company and can't post the solution now. However yes I believe it was TBF that I used and yes the router was a fast xeon server. – hookenz Jan 08 '19 at 05:58
  • Note that 2 Gbps traffic requires handling minimum of 130000 packets per second (if every packet is max size) and in practice somewhere around half a million packets per second. If you network adapter require hardware interrupt per packet and only supports one CPU core the changes are high that it will not scale. A good NIC is required and you may need to use Linux kernel with at least PREEMPT enabled. – Mikko Rantalainen Aug 19 '20 at 10:50
  • 1
    I did manage to get this to work but unfortunatrly don't have the scripts that made it all work. I. Was using quite a powerful cpu though. – hookenz Aug 19 '20 at 12:09