When i use an openvpn (UDP connection) VPN, then a tcp connecting saturating the link (i.e. big download in firefox) causes a massive packetloss for all other connections. TCP, UDP and simple ICMP echo (ping) requests lose packets in a way, that even TCP connections time out.
For connections going though a non-vpn interface i do not have such problems, if there are several big downloads, they share the bandwith fairly. Only on the vpn link the connections are "unfair" scheduled.
I do not think its a server side problem, as the server should not do any prioritizing.
typical vpn packet (currently without much load on the vpn) on eth0:
16:06:50.228295 IP x.x.x.x.1149 > y.y.y.y.43560: UDP, length 145