HTB "rate" parameter limits available bandwidth

Question

Good day,

I have the following situation: 4 TCP streams of data from one machine to another. Each streams has its own destination TCP Port. 4 streams have different priorities : high, medium, low, bulk. High, medium, low generate 1.67Mbit/s and bulk generates 10Mbit/s. (iperf3 used to generate the traffic). Packets of each stream are marked with appropriate DiffServ mark (DSCP) and this mark is used for the classification of traffic in the HTB qdisc.

Goal: HTB qdisc should be configured in such way that at any time the high prio stream gets its requiered 1.67Mbit/s, medium prio is also guaranteed 1.67Mbit/s but with slightly lower prio and the rest of the traffic should be guaranteed 50kbit/s. Each stream must be able to use the whole link if it is idle and the stream generates more bandwidth than initially specified.

Generation of traffic:

High priority:
iperf3 -c 192.168.88.254 -p 5150 -t 62 -b 1.67M -l 128 -S 224 
Medium priority:
iperf3 -c 192.168.88.254 -p 5160 -t 62 -b 1.67M -l 4K -S 160 
Low priority:
iperf3 -c 192.168.88.254 -p 5170 -t 62 -b 1.67M -l 4K -S 96 
Bulk:
iperf3 -c 192.168.88.254 -p 5180 -t 62 -b 10M -l 4K -S 0 

Configuration of HTB qdisc

NI="eth2"
AC="sudo /sbin/tc class add dev "

# Delete previous qdiscs
sudo /sbin/tc qdisc del dev $NI root

# Add HTB as root with default class 40 for uncategorized traffic
sudo /sbin/tc qdisc add dev $NI root handle 1: htb default 40
sudo /sbin/tc class add dev $NI parent 1: classid 1:1 htb rate 3.5mbit ceil 1000mbit

# high priority stream DSCP 224 - 1110 0000 - 0xE0
$AC $NI parent 1:1 classid 1:10 htb rate 1.7mbit ceil 1000mbit prio 1
# medium priority stream 
$AC $NI parent 1:1 classid 1:20 htb rate 1.7mbit ceil 1000mbit prio 2
# low priority stream
$AC $NI parent 1:1 classid 1:30 htb rate 50kbit ceil 1000mbit prio 3
# bulk stream
$AC $NI parent 1:1 classid 1:40 htb rate 50kbit ceil 1000mbit prio 4 

# Add filters to classify packets based on dscp mark

# high priority DSCP 224 - 1110 0000 - 0xE0
sudo /sbin/tc filter add dev $NI protocol ip parent 1: prio 1 u32 match ip tos 0xE0 0xff flowid 1:10
# medium priority DSCP 160 - 1010 0000 - 0xA0
sudo /sbin/tc filter add dev $NI protocol ip parent 1: prio 2 u32 match ip tos 0xA0 0xff flowid 1:20
# low priority DSCP 96 - 1100 0000 - 0x60
sudo /sbin/tc filter add dev $NI protocol ip parent 1: prio 3 u32 match ip tos 0x60 0xff flowid 1:30
# bulk DSCP 0 - 0000 0000 - 0x00
sudo /sbin/tc filter add dev $NI protocol ip parent 1: prio 4 u32 match ip tos 0x00 0xff flowid 1:40

The traffic is classified correctly. I can see relevant counters in tc class statistics going up. I rechecked this multiple times.

Problem: This configuration correctly allocates bandwidth to high and medium prio streams. Low prio and bulk also receive their 50kbit. However, I am not able to push through the link more than the value specified in the root class1:1 as rate i.e. 3.5mbit.

In every article and manual about HTB that I read it was stated, that "rate" parameter is the minimal guaranteed rate to the class and "ceil" is the max amount it can get. In my case it seems that "rate" caps the link at specified value. This is definitely not the desired and expected behaviour.

If I set the "rate" parameter of the root class to same value as "ceil" i.e. 1000mbit, no prioritisation takes place and the available bandwidth is divided equally between all flows. This is not the desired behavior as in the case of fluctuations of available bandwidth, prio traffic will get less than 1.67Mbit/s

Have I misunderstood the meaning of "rate" parameter in the root class? Is this problem somehow related to onther HTB parameters like "quantum"? I also observed that every class has negative amount of tokens during the transmission of data. Is this bad? If so, what parameters should i tune and how?

Thank You in advance!

Answer 1

The ceil parameter had no effect in the root class, you should only set the rate one.

If your goal is to make the ceiling part of child classes scale, you can use hfsc instead of HTB but it won't prevent from asking for the max rate even if you have not enough bandwidth.

If your goal is to deal with a fluctuent internet connection, it won't work this way because you are limiting the upload part, and the download one still won't be prioritized. You can search for ifb to limit the download bandwidth.

In any case, tc won't be able to detect your real available bandwidth. Be sure to use fq_codel though, for a better queue management, bbr for TCP, and a recent Linux kernel, so you limit the impact of being able to ask for more bandwidth you have.