1

I hope you can help.

I currently have haproxy 1.7.9 compiled with Tproxy on centos 7.

This is used to load balance 6 proxy servers for internet access on the back end. It is setup as a one arm nat with a 10gb vmnet3 nic. The 6 proxy servers are cisco WSA. keepalived is setup with a vip. IPtables is utilized to encapsulate the traffic to ensure the back end cisco wsa proxy receives the source ip address rather than the haproxy ip address.

We are getting performance/latency issues with users reporting slow internet access etc, this is since the user count has increased, can anyone see or suggest anything on the load balancer that may be causing issues?

config:

global
    daemon
    log 127.0.0.1 local4
    maxconn 40000
    ulimit-n 81000

defaults
    log global
    timeout connect 5000
    timeout client 300000
    timeout server 300000
    mode http

frontend http-in
    bind *:80,*:443,*:8021
    maxconn 50000
    default_backend backend_servers
    option  forwardfor
    #option http-server-close
    #option forceclose

backend backend_servers
    #option http-server-close
    #option forceclose
    balance leastconn
    stick-table type ip size 20240k expire 5m
    stick on src
    server  edu-wsa01 10.80.10.111:80 check inter 30000 fall 3
    server  edu-wsa02 10.80.10.112:80 check inter 30000 fall 3
    server  edu-wsa03 10.80.10.113:80 check inter 30000 fall 3
    server  edu-wsa04 10.80.10.114:80 check inter 30000 fall 3
    server  edu-wsa06 10.80.10.116:80 check inter 30000 fall 3
    server  edu-wsa07 10.80.10.117:80 check inter 30000 fall 3
    #server  web2 10.80.10.248:80 check inter 30000 fall 3
    #server  web3 10.80.10.247:80 check
    source 0.0.0.0 usesrc clientip

Below this is just stats config on port 7000

I was looking to do direct server return however i am having issues with the WSA supporting loopback.

chicks
  • 3,639
  • 10
  • 26
  • 36
Rebus
  • 86
  • 8
  • What are loads on the load balancer? Is the memory free? How many cores do you have? – Jaroslav Kucera Nov 28 '17 at 12:38
  • It has 1 core and from top command cpu on around 20-25% and 4GB ram using about 300-400mb most of the time. – Rebus Nov 28 '17 at 13:25
  • And the load is less than 1? Have you applied some network performance tuning? – Jaroslav Kucera Nov 28 '17 at 13:30
  • load average: 0.11, 0.17, 0.15, no network performance tuning has been applied – Rebus Nov 28 '17 at 13:32
  • Do you have at least `tuned` runing with `network-latency` performance profile? – Jaroslav Kucera Nov 28 '17 at 13:35
  • no, what would you suggest is my best next step? – Rebus Nov 28 '17 at 13:41
  • Set the tuned: `tuned-adm profile network-latency` – Jaroslav Kucera Nov 28 '17 at 13:49
  • Thanks for that, its been set but im still seeing a lot of latency, 4 chk fails an average of around 500 con errors per backend device and around 30k warning retr per backend device. – Rebus Nov 28 '17 at 15:58
  • As you speak about vmnet3, it's VMware. Can you assign additional core so it would solve interrupts. You should pin the haproxy process to the second core then, as kernel usually assigns interrupts to the first core. Also you can check your kernel parameters related to networking buffers. Defaults are sufficient for 1Gb NIC, for 10 Gb NICs it should be enlarged - darksideclouds.wordpress.com/2016/10/10/tuning-10gb-nics-highway-to-hell/ Take these values as hint, for example I'm not sure about the congestion control htcp (may/may not help you in comparison to cubic) – Jaroslav Kucera Nov 28 '17 at 17:45
  • If you succeed, please share solution applied in the answer so we have confirmed howto. – Jaroslav Kucera Nov 29 '17 at 08:49
  • Hi, i believe i have the networking sorted now i just need to bind the processors in haproxy. i am using nbproc 4 then cpu-map 1 1, cpu-map 2 2 and cpu-map 3 3 with stats bind-process 3 and frontend bind-process 1, backend bind-process 2, this is to ensure vmxnet3 has cpu core 0 free for use. however haproxy doesnt seem to want to start, i have googled to see if syntax has changed in 1.7.9 but cant seem to find anything concrete, any thoughts? – Rebus Dec 04 '17 at 13:06

0 Answers0