Backend fetch failed - varnish wont fetch drupal backend

Question

I run a drupal 7 application(3backends) and i have 3 varnish servers continuously denying to fetch backend. I have read many similar error here but still cant solve my problem, throwing 503 varnish fetch failed guru meditation. I have read all the posts here, and all seemed to recommend high timeout, i have set to 600s, and many does not recommend .probe, I have round.robin (switching the backend) and here is my backend config:

backend project1 {
.host = "myhost.ip";
.port = "80";
.connect_timeout = 600s;
.first_byte_timeout = 600s;

.probe = {
.timeout = 600s;
.interval = 10s;
.window = 5;
.threshold = 2;
.request =
"GET HTTP/1.1"
"Host: example.com"
"Connection: close";
}
}

i have monitored my error log and access log, by the way and i noticed i have too much errors like this but i dont want you to be biased.

[info] Client prematurely closed connection (broken pipe)

and sometimes

reqv failed

As a sidenote i want to mentionalso, i have fast-cgi error still ongoing but i dont think it has relationship with my varnish error:

Primary Script unknown  ......, fastcgi, upstream:127.0.0.1

i run nginx+php-fpm via fast-cgi. i am really unsure, what config varnish is expecting from backend that make it refuse to fetch it,

It happened that a wrong nginx config gave me this error before so here is my nginx config:

user nginx nginx;
worker_processes 4;

error_log /var/log/nginx/error.log info;

events {
    worker_connections 8192;
    multi_accept    on;
    use epoll;
}
worker_rlimit_nofile 64000;
http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format main
            '$remote_addr - $remote_user [$time_local] '
            '"$request" $status $bytes_sent '
            '"$http_referer" "$http_user_agent" ';

    client_header_timeout 10m;
    client_max_body_size 100m;
    client_body_timeout 10m;
    send_timeout 10m;
    client_body_buffer_size 3m;
    connection_pool_size 256;
    client_header_buffer_size 1k;
    large_client_header_buffers 4 2k;
    request_pool_size 32k;

    gzip            on;
    gzip_vary on;
    gzip_disable "MSIE [1-6]\.";
    gzip_min_length 10240;
    gzip_proxied    expired no-cache no-store private auth;
    gzip_types      text/plain application/xml;

    open_file_cache         max=2000 inactive=20s;
    open_file_cache_valid   60s;
    open_file_cache_min_uses        5;
    open_file_cache_errors          off;

    output_buffers 1 32k;
    postpone_output 1460;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    fastcgi_send_timeout 1800;
    fastcgi_read_timeout 1800;
    fastcgi_connect_timeout 1800;
    fastcgi_ignore_client_abort on;
    keepalive_timeout 75 20;
    ignore_invalid_headers on;
    index index.html;

    server {
            listen 80;
            server_name  example.com;
            rewrite ^(.*) http://example.com$1 permanent;
            }
    server {
            listen 80;
            server_name www.example.com;
            access_log /var/log/nginx/access.log main;
            error_log /var/log/nginx/error.log info;              
            root /var/www/public;
            index index.php index.phtml index.html;
            autoindex on;

            gzip_types text/plain text/css application/json application/x-ja
vascript text/xml application/xml application/xml+rss text/javascript applicatio
n/javascript;

            location ~ \..*/*\.php$ {
                    return 403;
            }
            location ~^/sites/.*/private/{
                    return 403;
            }
            location ~^/sites/.*/files/* {
                    try_files $uri @rewrite;
            }
            location ~ (^|/)\. {
                    return 403;
            }

            location / {

                    try_files $uri @rewrite;
                    }
            location @rewrite {
                    rewrite ^ /index.php;

            }
            location ~ \.php$ {
                    try_files $uri @rewrite;
                    #fastcgi_pass unix:/var/run/php5-fpm.sock;
                    fastcgi_pass 127.0.0.1:9000;
                    fastcgi_index  index.php;
             fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
                    fastcgi_buffer_size 128k;
                    fastcgi_buffers 256 16k;
                    fastcgi_busy_buffers_size 256k;
                    fastcgi_temp_file_write_size 256k;
                    fastcgi_keep_conn       off;
                    include /etc/nginx/fastcgi_params;

                    }


            location ~* \.(jpg|jpeg|css|gif|png|js|ico|xml)$ {

                    try_files $uri $uri/ ;
                    access_log off;
                    log_not_found   off;
                    expires 30d;
                    }

}

Can someone point me which direction to debug this issue? i am likely sure, pb is on my backend rather than my varnish but unsure, why the website sometimes loads, and sometimes throw a straight/awfull "backend fetch failed 503 - guru meditation". thanks for your precious help.

UPDATE:

the fix was to disable gzip. I had an empty content and redirect on homepage so, gzip+ header-content-length=0 (empty) triggered a red flag to varnish, varnish labeled as unhealthy to compress something with size 0. Wrong header. either disable gzip or return some content to server response would fix this issue

Answer 1

In your .request you don't seem to have specified a document. I assume it's therefore trying to load your entire Drupal site as a probe. You could change this to:

"GET /sitehealth.html HTTP/1.1"

where sitehealth.html is just a simple text file that the probe can load.

Further troubleshooting, can you disable probes entirely and see if that makes a difference? Do you get any errors when hitting the website directly, without going through the cache?

Basically, get the very simplest configuration that reliably works and then add your additional features one at a time until you hit something that breaks.