Rolling update behind HAProxy

Question

I am using an infrastructure platform consisting of docker containers running on a CentOS host.

Scaling multiple containers through docker compose.
Consul for service discovery.
Registrator for service registry
HAProxy as a load balancer
Consul-template for generating haproxy configuration each time the container is added/removed.

Each time we update our build, new containers are spawned and old containers are removed. The problem is containers are removed instantaneously without regards to the existing requests being served by the containers. Ideally, I should have a signal to mark the containers in maintenance or drain mode so that existing connections can be served and then gracefully removed from haproxy first and then physically.

Has anyone faced this problem before or done the similar autoscaling workflow through containers.

So each time you scale, all containers are removed and then new containers are brought up? — Will Barnwell, Jun 27 '16 at 20:54
Not scale, when I deploy a new version of the container @WillBarnwell — Madhur Ahuja, Jun 28 '16 at 13:09
And you want to not immediatly kill the previous running set but instead let them gracefully quit? — Will Barnwell, Jun 28 '16 at 13:43
That is right .. essentially because previous ones will be probably still serving requests @WillBarnwell — Madhur Ahuja, Jun 28 '16 at 16:40
`Each time we update our build, new containers are spawned and old containers are removed` How? — 030, Jun 30 '16 at 23:51

score 3 · Accepted Answer · answered Jun 28 '16 at 17:23

From my understanding of what you're asking, you have control over all the steps of this process and want to bring up new versions of a containerized application which sit behind HAProxy without interrupting client connections to the old applications, but preventing new connections.

This seems like a good use of the HAProxy disable server command.

You should be able to disable your deprecated servers, marking them as down for maintenance in HAProxy, which should gracefully allow clients to finish their business, then your consul/HAProxy magic can bring the new servers online.

Rolling update behind HAProxy

1 Answers1