3

This question is related to another question with a great answer and script from @Oliver.

The goal: I want to modify/extend the script provided in this answer to suit my requirements, which are as follows:

  1. I have a large number of clients (up to 1000). Each client shall be assigned a subscription class and corresponding maximum data rate based on its CN (common-name). These rate limits shall be applied when the client connects and shall be removed when it disconnects:

    • bronze: 1 mbit
    • silver: 10 mbit
    • gold: 100 mbit
  2. I would like to adjust each client’s subscription class and corresponding active data rate limit on the fly, while the client is connected to the OpenVPN server. The client should not have to reconnect to the OpenVPN server. Is this possible or do we have to disconnect and reconnect each client to OpenVPN to cause the script to be called again to change the tc configuration?

  3. Instead of modifying the tc configuration manually using the shell, how would we update the client subscription class and corresponding active data rate limit on the fly from another computer or application (i.e. via PHP)?

Many thanks

Server Programmer
  • 237
  • 2
  • 4
  • 11

1 Answers1

10

Here is a solution, how to do traffic shaping for data rate limiting of individual clients with tc (traffic control) using a script called by OpenVPN.

The traffic control settings are handled in a script tc.sh with the following features:

  • Called by OpenVPN using directives: up, down, client-connect and client-disconnect
  • All settings are passed via environment variables
  • Supports theoretically up to /16 subnets (up to 65534 clients)
  • Filtering using hashing filters for very fast massive filtering
  • Filters and classes are set only for clients currently connected, and are individually added and removed without affecting other tc settings using unique identifiers (hashtables, handles, classids). These identifiers are generated from the last 16 bits of the client's remote vpn IP
  • Individual limiting/throttling of clients based on CN-name (client certificate common name)
  • Client settings are stored in files containing their "subscription class" (bronze, silver and gold), to use other classes simply edit the script and modify as needed.
  • "Subscription class" and the corresponding data rate ("bandwidth") can be modified on the fly from external applications while a client is connected.

Configuration

OpenVPN server configuration /etc/openvpn/tc/conf:

port 1194
proto udp
dev tun
sndbuf 0
rcvbuf 0
ca ca.crt
cert server.crt
key server.key
dh dh.pem
tls-auth ta.key 0
topology subnet
server 10.8.0.0 255.255.0.0
keepalive 10 60
comp-lzo
persist-key
persist-tun
status /var/log/openvpn-tc-status.log
log /var/log/openvpn-tc.log
verb 3
script-security 2
down-pre
up /etc/openvpn/tc/tc.sh
down /etc/openvpn/tc/tc.sh
client-connect /etc/openvpn/tc/tc.sh
client-disconnect /etc/openvpn/tc/tc.sh
push "redirect-gateway def1"
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"

Replace the DNS servers in the last 2 lines with the correct IP addresses.

Traffic control script /etc/openvpn/tc/tc.sh:

#!/bin/bash

ipdir=/etc/openvpn/tc/ip
dbdir=/etc/openvpn/tc/db
ip="$ifconfig_pool_remote_ip"
cn="$common_name"
ip_local="$ifconfig_local"

debug=0
log=/tmp/tc.log

if [[ "$debug" > 0 ]]; then
  exec >>"$log" 2>&1
  chmod 666 "$log" 2>/dev/null
  if [[ "$debug" > 1 ]]; then
    date
    id
    echo "PATH=$PATH"
    [[ "$debug" > 2 ]] && printenv
  fi
  echo
  echo "script_type=$script_type"
  echo "dev=$dev"
  echo "ip=$ip"
  echo "user=$cn"
  echo "\$1=$1"
  echo "\$2=$2"
  echo "\$3=$3"
fi

cut_ip_local() {
  if [ -n "$ip_local" ]; then
    ip_local_byte1=`echo "$ip_local" | cut -d. -f1`
    ip_local_byte2=`echo "$ip_local" | cut -d. -f2`
  fi

  [[ "$debug" > 0 ]] && echo "ip_local_byte1=$ip_local_byte1"
  [[ "$debug" > 0 ]] && echo "ip_local_byte2=$ip_local_byte2"
}

create_identifiers() {
  if [ -n "$ip" ]; then
    ip_byte3=`echo "$ip" | cut -d. -f3`
    handle=`printf "%x\n" "$ip_byte3"`
    ip_byte4=`echo "$ip" | cut -d. -f4`
    hash=`printf "%x\n" "$ip_byte4"`
    classid=`printf "%x\n" $((256*ip_byte3+ip_byte4))`
  fi

  [[ "$debug" > 0 ]] && echo "ip_byte3=$ip_byte3"
  [[ "$debug" > 0 ]] && echo "ip_byte4=$ip_byte4"
  [[ "$debug" > 0 ]] && echo "handle=$handle"
  [[ "$debug" > 0 ]] && echo "hash=$hash"
}

start_tc() {
  [[ "$debug" > 1 ]] && echo "start_tc()"

  cut_ip_local

  echo "$dev" > "$ipdir"/dev

  tc qdisc add dev "$dev" root handle 1: htb
  tc qdisc add dev "$dev" handle ffff: ingress

  tc filter add dev "$dev" parent 1:0 prio 1 protocol ip u32
  tc filter add dev "$dev" parent 1:0 prio 1 handle 2: protocol ip u32 divisor 256
  tc filter add dev "$dev" parent 1:0 prio 1 protocol ip u32 ht 800:: \
      match ip dst "${ip_local_byte1}"."${ip_local_byte2}".0.0/16 \
      hashkey mask 0x000000ff at 16 link 2:

  tc filter add dev "$dev" parent ffff:0 prio 1 protocol ip u32
  tc filter add dev "$dev" parent ffff:0 prio 1 handle 3: protocol ip u32 divisor 256
  tc filter add dev "$dev" parent ffff:0 prio 1 protocol ip u32 ht 800:: \
      match ip src "${ip_local_byte1}"."${ip_local_byte2}".0.0/16 \
      hashkey mask 0x000000ff at 12 link 3:
}

stop_tc() {
  [[ "$debug" > 1 ]] && echo "stop_tc()"

  tc qdisc del dev "$dev" root
  tc qdisc del dev "$dev" handle ffff: ingress

  [ -e "$ipdir"/dev ] && rm "$ipdir"/dev
}

function bwlimit-enable() {
  [[ "$debug" > 1 ]] && echo "bwlimit-enable()"

  create_identifiers

  echo "$ip" > "$ipdir"/"$cn".ip

  # Find this user's bandwidth limit
  [[ "$debug" > 0 ]] && echo "userdbfile=${dbdir}/${cn}"
  user=`cat "${dbdir}/${cn}"`
  [[ "$debug" > 0 ]] && echo "subscription=$user"

  if [ "$user" == "gold" ]; then
    downrate=100mbit
    uprate=100mbit
  elif [ "$user" == "silver" ]; then
    downrate=10mbit
    uprate=10mbit
  elif [ "$user" == "bronze" ]; then
    downrate=1mbit
    uprate=1mbit
  else
    downrate=10kbit
    uprate=10kbit
  fi

  # Limit traffic from VPN server to client
  tc class add dev "$dev" parent 1: classid 1:"$classid" htb rate "$downrate"
  tc filter add dev "$dev" parent 1:0 protocol ip prio 1 \
      handle 2:"${hash}":"${handle}" \
      u32 ht 2:"${hash}": match ip dst "$ip"/32 flowid 1:"$classid"

  # Limit traffic from client to VPN server
  # Maybe better use ifb for ingress? See: https://serverfault.com/a/386791/209089
  tc filter add dev "$dev" parent ffff:0 protocol ip prio 1 \
      handle 3:"${hash}":"${handle}" \
      u32 ht 3:"${hash}": match ip src "$ip"/32 \
      police rate "$uprate" burst 80k drop flowid :"$classid"
}

function bwlimit-disable() {
  [[ "$debug" > 1 ]] && echo "bwlimit-disable()"

  create_identifiers

  tc filter del dev "$dev" parent 1:0 protocol ip prio 1 \
      handle 2:"${hash}":"${handle}" u32 ht 2:"${hash}":
  tc class del dev "$dev" classid 1:"$classid"
  tc filter del dev "$dev" parent ffff:0 protocol ip prio 1 \
      handle 3:"${hash}":"${handle}" u32 ht 3:"${hash}":

  # Remove .ip
  [ -e "$ipdir"/"$cn".ip ] && rm "$ipdir"/"$cn".ip
}

case "$script_type" in
  up)
    start_tc
    ;;
  down)
    stop_tc
    ;;
  client-connect)
    bwlimit-enable
    ;;
  client-disconnect)
    bwlimit-disable
    ;;
  *)
    case "$1" in
      update)
        [ -z "$2" ] && echo "$0 $1: missing argument [client-CN]" >&2 && exit 1
        [ ! -e "$ipdir"/"$2".ip ] &&  \
            echo "$0 $1 $2: file $ipdir/$2.ip not found" >&2 && exit 1
        [ ! -e "$ipdir"/dev ] && \
            echo "$0 $1: file $ipdir/dev not found" >&2 && exit 1
        ip=`cat "$ipdir/$2.ip"`
        dev=`cat "$ipdir/dev"`
        cn="$2"
        bwlimit-disable
        bwlimit-enable
        ;;
      *)
        echo "$0: unknown operation [$1]" >&2
        exit 1
        ;;
    esac
    ;;
esac

exit 0

Make it executable:

chmod +x /etc/openvpn/tc/tc.sh

Subscription database directory /etc/openvpn/tc/db/:

This directory contains a file per client named after its CN-name containing the "subscription class" string, configure as follows:

mkdir -p /etc/openvpn/tc/db
echo bronze > /etc/openvpn/tc/db/client1
echo silver > /etc/openvpn/tc/db/client2
echo gold > /etc/openvpn/tc/db/client3

IP database directory /etc/openvpn/tc/ip/:

This directory will contain the CN-name <-> IP-address relation and the tun interface during run-time, which has to be provided for an external application updating the tc settings while clients are connected.

mkdir -p /etc/openvpn/tc/ip

It will look as follows:

root@ubuntu:/etc/openvpn/tc/ip# ls -l
-rw-r--r-- 1 root root    9 Jun  1 08:31 client1.ip
-rw-r--r-- 1 root root    9 Jun  1 08:30 client2.ip
-rw-r--r-- 1 root root    9 Jun  1 08:30 client3.ip
-rw-r--r-- 1 root root    5 Jun  1 08:25 dev
root@ubuntu:/etc/openvpn/tc/ip# cat *
10.8.0.2
10.8.1.0
10.8.2.123
tun0

Enable IP forwarding:

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p

Configuring NAT (network address translation):

If you have a static external IP address use SNAT:

iptables -t nat -A POSTROUTING -s 10.8.0.0/16 -o <if> -j SNAT --to <ip>

Or if you have a dynamically assigned IP address use MASQUERADE (slower):

iptables -t nat -A POSTROUTING -s 10.8.0.0/16 -o <if> -j MASQUERADE

while

  • <if> is the name of the external interface (i.e. eth0)
  • <ip> is the IP address of the external interface

Script usage and showing tc configuration

Updating "subscription class" and tc settings from external application:

While the OpenVPN server is up and the client connected issue the following commands (example to upgrade client1 to "gold" subscription):

echo gold > /etc/openvpn/tc/db/client1
/etc/openvpn/tc/tc.sh update client1

tc commands to show the settings:

tc -s qdisc show dev tun0
tc class show dev tun0
tc filter show dev tun0

Additional information

Notes and possible optimizations:

  • The script and tc settings were only tested using a small number of clients
  • Large scale testing with massive simultaneous client traffic has to be done and possibly the tc settings have to be optimized
  • I do not completely understand how the ingress settings work. They should probably be optimized with the use of ifb interface as explained in this answer.

Related documentation for a deeper understanding:

rda
  • 1,887
  • 1
  • 12
  • 20
  • Does anybody know how to show the `ingress` settings? – rda Jun 01 '16 at 10:48
  • "Filters and classes are set only for clients currently connected", Is this a limitation of TC for individual bandwidth control ? I am trying to control bandwidth on ikev2 vpn server. Banwidth control only works for the clients that are connected already before running script. I m using this : https://linuxscriptshub.com/bandwidth-control-on-ikev2-with-tc-and-iptables/ – Ajji May 09 '18 at 13:22
  • @Ajji No, this is not a limitation of TC, but a limitation of the script in this answer. When a client connects via OpenVPN a TC class for that client is created with the respective rate limit. Once the client disconnects, the class is removed. – rda May 14 '18 at 13:37