!Important. SOLUTION FOUND. Everything below is working fine for me. And this is almost the only case when you may copy solution from the question. For it read "My Configuration" section
Main Question
On an attempt to open the homepage (through https 443 or http 80 does not matter) I see 503 Varnish error. It looks like Varnish cannot access Nginx back.
Varnish Request Log
* << Request >> 20
- Begin req 19 rxreq
- Timestamp Start: 1587133057.263443 0.000000 0.000000
- Timestamp Req: 1587133057.263443 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.112.7 46616 a0
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.0
- ReqHeader X-Real-IP: 192.168.112.1
- ReqHeader X-Forwarded-For: 192.168.112.1
- ReqHeader X-Forwarded-Proto: https
- ReqHeader X-Forwarded-Port: 443
- ReqHeader Host: magento2.docker
- ReqHeader Connection: close
- ReqHeader Cache-Control: max-age=0
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
- ReqHeader Sec-Fetch-Site: none
- ReqHeader Sec-Fetch-Mode: navigate
- ReqHeader Sec-Fetch-User: ?1
- ReqHeader Sec-Fetch-Dest: document
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,be;q=0.6
- ReqHeader Cookie: mage-banners-cache-storage=%7B%7D; _ga=GA1.2.164324136.1586263205; private_content_version=b9dd22e11f7865017754892a623f70f0; mage-cache-storage=%7B%7D; mage-cache-storage-section-invalidation=%7B%7D; mage-messages=; recently_viewed_product=%7B%7D
- ReqUnset X-Forwarded-For: 192.168.112.1
- ReqHeader X-Forwarded-For: 192.168.112.1, 192.168.112.7
- VCL_call RECV
- VCL_return pass
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- Link bereq 21 pass
- Timestamp Fetch: 1587133057.263600 0.000157 0.000157
- RespProtocol HTTP/1.1
- RespStatus 503
- RespReason Backend fetch failed
- RespHeader Date: Fri, 17 Apr 2020 14:17:37 GMT
- RespHeader Server: Varnish
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- RespHeader X-Varnish: 20
- RespHeader Age: 0
- RespHeader Via: 1.1 varnish (Varnish/6.2)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1587133057.263609 0.000166 0.000009
- Filters
- RespHeader Content-Length: 279
- RespHeader Connection: close
- Timestamp Resp: 1587133057.263640 0.000197 0.000031
- ReqAcct 1164 0 1164 242 279 521
- End
* << BeReq >> 21
- Begin bereq 20 pass
- VCL_use boot
- Timestamp Start: 1587133057.263477 0.000000 0.000000
- BereqMethod GET
- BereqURL /
- BereqProtocol HTTP/1.0
- BereqHeader X-Real-IP: 192.168.112.1
- BereqHeader X-Forwarded-Proto: https
- BereqHeader X-Forwarded-Port: 443
- BereqHeader Host: magento2.docker
- BereqHeader Cache-Control: max-age=0
- BereqHeader Upgrade-Insecure-Requests: 1
- BereqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36
- BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
- BereqHeader Sec-Fetch-Site: none
- BereqHeader Sec-Fetch-Mode: navigate
- BereqHeader Sec-Fetch-User: ?1
- BereqHeader Sec-Fetch-Dest: document
- BereqHeader Accept-Encoding: gzip, deflate, br
- BereqHeader Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7,be;q=0.6
- BereqHeader Cookie: mage-banners-cache-storage=%7B%7D; _ga=GA1.2.164324136.1586263205; private_content_version=b9dd22e11f7865017754892a623f70f0; mage-cache-storage=%7B%7D; mage-cache-storage-section-invalidation=%7B%7D; mage-messages=; recently_viewed_product=%7B%7D
- BereqHeader X-Forwarded-For: 192.168.112.1, 192.168.112.7
- BereqProtocol HTTP/1.1
- BereqHeader X-Varnish: 21
- VCL_call BACKEND_FETCH
- VCL_return fetch
- FetchError backend default: fail errno 111 (Connection refused)
- Timestamp Beresp: 1587133057.263561 0.000084 0.000084
- Timestamp Error: 1587133057.263564 0.000087 0.000002
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Fri, 17 Apr 2020 14:17:37 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- BerespHeader Content-Type: text/html; charset=utf-8
- BerespHeader Retry-After: 5
- VCL_return deliver
- Storage malloc Transient
- Length 279
- BereqAcct 0 0 0 0 0 0
- End
My Configuration
Source code can be found here: https://github.com/zhartaunik/magento2-docker
Here are some necessary moments for history:
docker-compose.yml
version: "3"
services:
varnish:
container_name: varnish
build:
context: varnish/
ports:
- 6081:6081
nginx:
container_name: nginx
build:
context: nginx/
working_dir: /etc/nginx
ports:
# {internal_in_docker}:{external_from_browser}
- 80:80
- 443:443
- 8001:8001
volumes:
- ./magento:/var/www/magento
env_file:
- ./.env
...
nginx/etc/vhost.conf
upstream fastcgi_backend {
server magento:9000;
}
server {
listen 80;
listen 443 ssl;
server_name _;
ssl on;
ssl_certificate /etc/nginx/ssl/magento.crt;
ssl_certificate_key /etc/nginx/ssl/magento.key;
keepalive_timeout 300s;
location / {
proxy_pass http://varnish:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
proxy_read_timeout 600s;
}
}
server {
listen 8001;
server_name localhost;
set $MAGE_ROOT /var/www/magento;
set $MAGE_MODE developer;
I faced the issue because I didn't use the desired varnish.vcl. If you use this one, everything should work fine for you
varnish/etc/varnish.vcl
vcl 4.0;
import std;
# The minimal Varnish version is 4.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'SSL-OFFLOADED: https'
backend default {
.host = "nginx";
.port = "8001";
}
...
How in my understanding it should work
- We open through browser homepage: https://magento2.docker/
- As inside
docker-compose.ymlconfigured that Nginx is listening to 443 port we go to it'svhost.confsection (1st part) - Because of the row
proxy_pass http://varnish:6081;we are directing to varnish container through 6081 port. nginx_container> telnet varnish 6081returns SUCCESS- According to
docker-compose.ymlvarnish configured to listen 6081 (Is it really required to add this port to yml?) we walk tovarnish/etc/varnish.vcl(Could this file contain onlybackend defaultsection for debug purpose only?) - Inside
backend defaultwe see host=nginx, port=8001. This should direct us to Nginx.varnish_container> telnet Nginx 8001returns SUCCESS. However looks like directly here WE ARE NOT GOING to Nginx - The 2nd part of
vhost.confdoesn't start to work. If we try to open http://magento2.docker:8001/ Nginx resolves this host and application start working.
What I'm looking for here
- The solution to a problem
- Two bolded questions above
- Any suggestions how to debug? How to check from varnish container what does respond Nginx?
Update #1 (Solution was found)
I've tested your Docker setup and the main thing I notice is that the backend definition of /etc/varnish/default.vcl in your varnish container is the following: backend default { .host = "127.0.0.1"; .port = "8080"; }
I found a mistake in DockerFile. I used the default config with wrong host/port settings. I changed this value and everything started working fine.
Here are the sources (all of them you may find at https://github.com/zhartaunik/magento2-docker):
varnish/Dockerfile
FROM varnish:6.2
COPY etc/varnish.vcl /etc/varnish/
COPY bin/docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod +x /docker-entrypoint.sh
ENV VARNISH_CONFIG /etc/varnish/varnish.vcl
ENV VARNISH_STORAGE malloc,2048m
ENV VARNISH_LISTEN :6081
ENV VARNISH_MANAGEMENT_LISTEN 127.0.0.1:6082
EXPOSE 6081
EXPOSE 6082
ENTRYPOINT ["/bin/bash", "/docker-entrypoint.sh"]
varnish/bin/docker-entrypoint.sh
#!/bin/bash
set -e
exec varnishd \
-j unix,user=vcache \
-F \
-f ${VARNISH_CONFIG} \
-s ${VARNISH_STORAGE} \
-a ${VARNISH_LISTEN} \
-T ${VARNISH_MANAGEMENT_LISTEN} \
-p feature=+esi_ignore_https \
${VARNISH_DAEMON_OPTS}
I also removed depends_on section.
