HAProxy health check is logged as TCP_MISS/000 in Squid

Question

We have a HAProxy front-end, load balancing a layer of Squid servers, which handles data from an Apache ModSecurity layer, and which ultimately serves up data from IIS cluster. Each Squid server is mapped to a ModSecurity server in a 1:1 relationship (e.g. N squid servers and N ModSecurity servers).

One server in the cluster continually sees higher health check times in HAProxy. The check.txt file is configured to bypass cache, so it will retrieve the file from the IIS layer. The Squid access logs (example below) show a TCP_MISS/000 instead of a TCP_MISS/200 as all the "good" servers show. Has anyone seen this behavior before?

Also, in the "good servers", you'll see that DIRECT/10.xxx.xxx.221 is consistent. However, in the "bad server", the logs alternate between DIRECT/10.xxx.xxx.222 and DIRECT/www.example.com.

Thoughts?

Results from bad server:

1308159554.223      0 xxx.xxx.xxx.49 TCP_MISS/000 0 HEAD http://www.example.com/check.txt - DIRECT/www.example.com -
1308159558.526      1 xxx.xxx.xxx.86 TCP_MISS/000 0 HEAD http://www.example.com/check.txt - DIRECT/10.xxx.xxx.222 -
1308159558.527      0 xxx.xxx.xxx.87 TCP_MISS/000 0 HEAD http://www.example.com/check.txt - DIRECT/www.example.com -
1308159558.621      0 xxx.xxx.xxx.48 TCP_MISS/000 0 HEAD http://www.example.com/check.txt - DIRECT/www.example.com -

Results from good servers:

1308159578.289      2 xxx.xxx.xxx.48 TCP_MISS/200 430 HEAD http://www.example.com/check.txt - DIRECT/10.xxx.xxx.221 text/plain
1308159578.695      6 xxx.xxx.xxx.86 TCP_MISS/200 430 HEAD http://www.example.com/check.txt - DIRECT/10.xxx.xxx.221 text/plain
1308159579.087      4 xxx.xxx.xxx.87 TCP_MISS/200 430 HEAD http://www.example.com/check.txt - DIRECT/10.xxx.xxx.221 text/plain

score 0 · Accepted Answer · answered Jun 17 '11 at 17:54

I believe I have found the solution to my problem.

The first issue is that the Squid server that had issues was suffering from I/O issues due to how it was communicating with the SAN datastore.

The second issue, likely a symptom of the first issue, was that two other deprecated HAProxy servers were also running health checks on the Squid servers. However, in a I/O bottleneck-free environment, I don't think that would be a true cause, unless it was related to the different versions of HAProxy.

HAProxy health check is logged as TCP_MISS/000 in Squid

1 Answers1