7

My internet connection goes like this:

Internet <-128kbps link-> Cisco Router (Public IP) <-LAN-> Linux router/server (Public IP) <-LAN-> Regular PCs (Public IPs)

The Cisco router:

  • the first Public IP allocated to my institution (/29)
  • is programmed to send all packets through the Linux router

The Linux router

  • The second Public IP allocated to my institution
  • is programmed to forward packets between the regular PCs and the Cisco router
  • act as a server too (mail, web, etc.)

The regular PCs (4 of them):

  • The rest of the Public IPs
  • use the Linux router as the gateway

I enabled the iptables packet logging on the Linux router and sometimes and I find out that:

  • Some packets are big, bigger than 20KB. Is that normal? (yes, it is normal. These are not packets, these are IP datagrams as Some Guy kindly explained)
  • Too many times the transmitted data (out to the internet) was bigger than 16KB. For example in a particular second 10572 bytes went in (no problem), 63521 bytes went out (to the Cisco router). It would take 4 seconds at least to send that 64KB through the 128kbps link. Meanwhile the Linux router is sending more data to the Cisco router, clogging its buffers. Not good.

Now, how can I configure the Linux router to shape traffic in a way that:

  1. Keep transmission speed to the max when the traffic is between these regular PCs and the Linux server.
  2. Slow down traffic to the outside world to avoid clogging the “out” line, using all (or almost all) the bandwidth available (128 kbps). No more “>16KB out seconds” on the trace.
  3. Guarantee 24kbps out to each regular PC, 24 kbps to the Linux server at any time. (8bkps left for overhead if necessary). IOW, 5 (pseudo) “bands”, 24kbps each one.
  4. In case there is any PC not using its full band, fairly share the idle bandwidth among the rest of the transmitting PCs
  5. Give priority to certain packets (DNS lookups, control packets), take priority away from others (torrent!!!), INSIDE each band, and not affecting other bands.

I have already marked (using IP tables --set-xmark option) each outgoing packet for each PC:

  1. Linux router to the outside world, high prio
  2. Linux router to the outside world, normal prio
  3. Linux router to the outside world, low prio
  4. First regular PC to the outside world, high prio

... And so on.

Each incoming packet is also marked using this scheme starting from 16.

I’m sorry for this long question, but I have given up to set this up using the tc command, there is too little documentation about traffic shaping and I don’t know where to go next.

3 Answers3

4

Assuming eth0 is a 100mbit Ethernet connection to the Cisco Router, it should be something like this (Isn’t it?):

tc qdisc add dev eth0 root handle 1: htb default 2
# 100 mbps
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
# To LAN traffic
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 99000kbit ceil 100mbit
# IN traffic
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 120kbit
# OUT traffic
tc class add dev eth0 parent 1:1 classid 1:4 htb rate 120kbit

# IN “bands” (one for each PC)
tc class add dev eth0 parent 1:3 classid 1:10 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:11 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:12 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:13 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:14 htb rate 24kbit ceil 120kbit

# OUT “bands” (one for each PC)
tc class add dev eth0 parent 1:4 classid 1:15 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:16 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:17 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:18 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:19 htb rate 24kbit ceil 120kbit

Would get me something like this:

+-----------------------------------------------------------+
|                      100 mbits (1:1)                      |
+---------+------------------------+------------------------+
| 99mbits |   120 kbits In (1:3)   |  120 kbits Out(1:4)    |
+  (1:2)  +----+----+----+----+----+----+----+----+----+----+
+---------+ PC1| PC2| PC3| PC4| PC5| PC1| PC2| PC3| PC4| PC5|
          |1:10|1:11|1:12|1:13|1:14|1:15|1:16|1:17|1:18|1:19|
          +----+----+----+----+----+----+----+----+----+----+

And for each band:

# PC1, IN
tc qdisc add dev eth0 parent 1:10 handle 20: prio
tc qdisc add dev eth0 parent 20:1 handle 22: sfq perturb 10
tc qdisc add dev eth0 parent 20:2 handle 23: sfq perturb 10
tc qdisc add dev eth0 parent 20:3 handle 24: sfq perturb 10

# PC1, OUT
tc qdisc add dev eth0 parent 1:15 handle 21: prio
tc qdisc add dev eth0 parent 21:1 handle 25: sfq perturb 10
tc qdisc add dev eth0 parent 21:2 handle 26: sfq perturb 10
tc qdisc add dev eth0 parent 21:3 handle 27: sfq perturb 10

+--------------------++--------------------+
|       PC1 IN       ||      PC1 OUT       |
+--------------------++--------------------+
|     PRIO (20:0)    ||     PRIO (21:0)    |
|      |      |      ||      |      |      |
| Prio | Prio | Prio || Prio | Prio | Prio |
|   1  |   2  |   3  ||   1  |   2  |   3  |
|(20:1)|(20:2)|(20:3)||(21:1)|(21:2)|(21:3)|
+------+------+------++------+------+------+
|  SFQ |  SFQ |  SFQ ||  SFQ |  SFQ |  SFQ |
|(22:0)|(23:0)|(24:0)||(25:0)|(26:0)|(27:0)|
+------+------+------++------+------+------+

And so on.

The rules should be like this

# PC1, OUT
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 21:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 2 fw flowid 21:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 21:3

# PC1, IN
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 16 fw flowid 20:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 17 fw flowid 20:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 18 fw flowid 20:3

and so on.

Any suggestion, comments, etc? (I have no experience in the field)

  • Your bandwidth numbers are wrong. All network vendors, when saying 100 MBit/s, they actually mean 1000 kbit/s or 1000 * 1000 bit/s. However, tc (and this has historic reasons) uses 1024. So in tc 100 mbit are 1024 kbit and 1024 * 1024 bit. IOW you root class is set to 1024 * 1024 bit/s in your script, but a 100 MBit/s Ethernet interface can only handle 1000 * 1000 bit/s and thus your script will shape far from optimal. Same is true for all numbers you provide in kbit of course. – Mecki Jan 20 '10 at 22:13
  • What makes IN IN? and what makes OUT OUT? It seems they are all the same except the IN has an odd number as classid, and the OUT has an even number as classid. – Qian Chen Jul 22 '18 at 01:26
  • read the end of the question: "I have already marked (using IP tables --set-xmark option) each outgoing packet for each PC ... Each incoming packet is also marked using this scheme starting from 16." – Yanko Hernández Álvarez Jul 24 '18 at 19:01
2

take a look here - it's quite simple tutorial how to use htb qos mechanism. in essence you mark packets on iptables and then assign to different ques depending on mark.

alternatively you might look at hfsc, it might actually work better with such slow internet connection, although i never used it.

regarding packet size - i cannot imagine 20kB large packet. even jumbo-frames in ethernet are shorter [ 9kB ].

edit: as Some Guy explains - LEN is length of defragmented IP packet, not a size of frame on the wire.

pQd
  • 29,561
  • 5
  • 64
  • 106
  • packet size: I thought so. These are the log offending lines. Interestingly, they were sent in tandem. – Yanko Hernández Álvarez Sep 30 '09 at 17:30
  • Sep 29 13:21:27 server BANDWIDTH_OUT:IN= OUT=eth0 SRC=RouterIP DST=69.63.180.40 LEN=20324 TOS=0x00 PREC=0x00 TTL=64 ID=8337 DF PROTO=TCP SPT=35245 DPT=80 WINDOW=46 RES=0x00 ACK URGP=0 UID=31 GID=31 MARK=0x1 – Yanko Hernández Álvarez Sep 30 '09 at 17:31
  • Sep 29 13:21:27 server BANDWIDTH_OUT:IN= OUT=eth0 SRC=RouterIP DST=69.63.180.40 LEN=26116 TOS=0x00 PREC=0x00 TTL=64 ID=8351 DF PROTO=TCP SPT=35245 DPT=80 WINDOW=46 RES=0x00 ACK URGP=0 UID=31 GID=31 MARK=0x1 – Yanko Hernández Álvarez Sep 30 '09 at 17:31
  • 2
    Don't confuse Ethernet frames (which have a size limitation based on hardware) with IP datagrams, which can be up to 64k in size. The LEN field in your logs is showing the size of the IP datagrams, which is indicated in their header. That's the total size, which includes any and all fragments. – DictatorBob Sep 30 '09 at 19:58
  • @Some Guy - Thanks, confusion cleared ;-). Big IP datagrams then ;-) – Yanko Hernández Álvarez Oct 06 '09 at 16:30
-1

What kind of Cisco is it? You generally can do shaping on IOS as well, it may be a bit easier to configure than Linux tc.

Heath
  • 1,240
  • 9
  • 4