0

I've got an nginx container which ends up with a full disk after it's been running for about 10 days. So if a new version of the app isn't released, errors start to occur that look like;

2022/01/15 22:45:04 [crit] 13#13: *406812 mkdir() "/var/cache/nginx/uwsgi_temp/9/07" failed (28: No space left on device) while reading upstream...

2022/01/15 22:44:37 [crit] 13#13: *406820 pwritev() "/var/cache/nginx/client_temp/0000001078" failed (28: No space left on device)...

This happened over the Christmas break so I thought the ideal situation here is to have the container health check ensure that there is free disk space. I thought I had achieved that with this container setup (but clearly not);

FROM nginx:1.21.5-alpine-perl

RUN apk update && \
    apk add --no-cache dnsmasq supervisor curl

COPY ./config/supervisord.conf /etc/supervisor/conf.d/supervisord.conf
COPY ./config/nginx.conf /etc/nginx/nginx.conf

HEALTHCHECK --interval=15s --timeout=30s \
    CMD exit $(( $(df / | tail -n1 | awk '{print $5}' | sed 's/\%//') > 95 ? 1 : 0 )) || exit 1

How should I check for disk space in the health check?

markwalker_
  • 131
  • 1
  • 2
  • 9

2 Answers2

1

You should check your nginx configuration and make sure its cache size limits are configured properly according to the space you have available in the container.

Tero Kilkanen
  • 34,499
  • 3
  • 38
  • 58
  • The config doesn't actually specify any cache settings. From what I can see, `proxy_cache_path` is what I'd need to define. I've also read `client_body_buffer_size` (not currently defined) should match `client_max_body_size` which is currently 20mb – markwalker_ Jan 16 '22 at 21:59
1

Enter the container and check which directory inside it occupies the space using du (my first guess would be log files). Check if you can change the nginx configuration in a way that the container doesn't fill up the filesystem. If not, mount an external volume in the problematic directory.

Gerald Schneider
  • 19,757
  • 8
  • 52
  • 79
  • That'd be ideal, but these containers are running on cloud formation stacks as fargate tasks. As far as I'm aware, it's not possible to connect to them. – markwalker_ Jan 16 '22 at 21:58