12

I'm currently working on a traffic shaping solution for ISP-level companies, and came to an interesting (kindof philosophical) problem.

Looking about the number of endpoints the system should handle (which is around ~20k) I got a little worried what would happen when I'd need to policy/shape traffic of more users. As I am currently using HFSC shaping tree (see tc-hfsc, mostly the same-but-cooler thing as better-known HTB) for whole network, I'd need to use more ClassIDs (obviously at least one for each user on the network). The problem which I found was that TC ClassID's are kindof limited - they are 16-bit numbers, which gives me a possible maximum of 64k users shaped by this solution.

Similarly, if I want to manage TC filters efficiently (e.g. not using the 'flush all technique'), I need to be able to delete or modify individual filter entries. (I'm using something similar to hash table from LARTC [1]). Again, the only method that seems to be working with this is to number all the filters using individual priorities (tc filter add dev ... prio 1). There's no other parameter that could be used for this purpose, and, regretably, prio is 16-bit-only as well.

My question is following: Does there exist some good method for enlarging the available "identifier space", such as 32-bit clsid's for 'tc class' command, and 32-bit priorities (or any other modification handles) for 'tc filter' command?

Thanks very much,

-mk

(btw I hope this will not go to "64k users should be enough for everyone" scenario...)

exa
  • 571
  • 4
  • 14
  • All those values are stored in kernel space, to make them larger you'd need to recompile your kernel and userspace utilities. Have you tried using 64bit kernel? They may be defined as 32bit there. – Hubert Kario Aug 21 '11 at 16:44
  • 64bit kernel uses the same sizes. For example, the filter number is u32-integer that consists of upper (protocol) and lower part (prio) both obviously 16bit. Class IDs are hardcoded as u16. Probably gonna try asking someone on LKML. – exa Aug 25 '11 at 15:52
  • 1
    even using hash for your filters, you gonna have a lot of IO problems if you are using that much filters (i guess for upstream and for downstream). I've spent a lot of time and had to patch the queues implementation inside kernel to have things working withou ksoftirqd. I used a patch from a guy called Simon Lodal that i met on LARTC 4 years ago. Take a look at his patch http://www.mail-archive.com/lartc@mailman.ds9a.nl/msg16279.html . You can try to send him an e-mail because he always has a very updated version (against the last kernel) with him. – Pabluez Dec 30 '11 at 14:05
  • @Pabluez Thanks very much, I will try to get the best out of it. – exa Dec 31 '11 at 14:50
  • 1
    I think your requirement is valid but as Pabluez wrote this certainly involves a lot of kernel changes. I don't want to say "you're doing it wrong", but I would encourage you to check out openflow, where the lower parts of the packet handling is done on the switch level and the policing is done in custom software, presumably running in user-space. I don't know if it fits your requirement, but it's certainly worth investigating. – AndreasM Jan 07 '12 at 10:02

2 Answers2

3

i think that you should not put 64k users with upstream and downstream classes and filters for each of them on the same interface. You can repeat the handlers for each interface you have, so add more interfaces. You will need a incredible work/server/NIC to have this things. If the server crashes, you will have 64k users offline (and it will crash easily with that amount of traffic). Don't forget that EACH packet that goes through your network card will be checked and classified by a filter and sent to a class to be queued. This is a lot of work for a NIC of an ISP gateway with 64k customers. Mainly with all the video stream that we have nowadays (which is hard to queue properly).

Pabluez
  • 131
  • 4
  • I'm ensuring service availability on some other level, but thanks for concern. Actually, with good NIC's, it isn't that hard to have a linux router that can forward 10Gbits. – exa Dec 31 '11 at 14:42
  • For original question, I was more interested in stuff like adding 5 different classes for each user, which would allow me to do a really good QOS job (like handling streams and realtime traffic separately), but is mostly unthinkable in current conditions (with my current use case of ~20k endpoints i'd be already behind the limit). – exa Dec 31 '11 at 14:48
  • 1
    okay, to forward 10Gbits is not any problem, the problem is having many 64k * 2(ups downs) filters and classes. Good luck though :D – Pabluez Jan 02 '12 at 11:52
0

You can split the traffic handling into two machines (using a third one) instead of handling all traffic on one machine. The traffic can be routed simply based on the source IP address. So, you will have optimally 10k users if you can divide the IP range(s) evenly.

Of course, you can use more than two machines if needed. I think this may be better than patching the Linux kernel and doing some other hacks. In short, the traffic shaping will be distributed on several machines. The central node will just forward the traffic to the right processing node.

Khaled
  • 35,688
  • 8
  • 69
  • 98