1

Given a set of web servers subject to rolling update, such as through kubernetes rolling updates, if a request is issued to one such termination-pending web server milliseconds before a SIGTERM signal is issued to said web server,

  1. Should the server signal the client that it is being SIGTERM'd and tell the client to "try again" using a different (or same) network address (with a potential delay)?
  2. Else, could the server redirect automatically the request to another pod/instance of the webserver that has been rolled up already?
  3. In the specific case of kubernetes, could the request be sent back to the service and let it know to send it back once at least one of the pods has been rolled out?
Philippe Hebert
  • 121
  • 3

1 Answers1

2

When a pod is terminating it has some time (by default 30 seconds) to complete the request when it receives SIGTERM and before it gets SIGKILL. You can configure longer timeouts. There is also preStop hook is called before SIGTERM is sent to a pod. See Kubernetes best practices: terminating with grace blog post for details.

Alternatively, you can configure load balancers to retry failed requests but this work only for idempotent requests.

AlexD
  • 8,179
  • 2
  • 28
  • 38
  • Thanks for the answer, @AlexD! I was aware that incoming network activity is stopped up to `terminationGracePeriodSeconds` seconds before the issuing of `SIGTERM`, which is by default `30s`. This indeed covers most use cases as it's really rare to have requests longer than `30s`. This being said, in the case where the request is a long-winded request and it cannot resolve in said amount of time, what would you suggest the strategy be? You can assume that the request is bounded by a transaction / is indempotent when returned early. – Philippe Hebert Dec 24 '21 at 16:59
  • @PhilippeHebert it really depends. What is the cost of failing the request? What options do we have to retry this request? How long can we delay the request and what is the cost of the delaying of the successful completion of the request? Too many variables. – AlexD Dec 24 '21 at 19:26
  • Those are very relevant questions; I suggest you add them as part of your answer, alongside "Is the request idempotent?". For the sake of completeness, if the handling of these requests can be categorized in a few approaches, I would also suggest describing these approaches as well. – Philippe Hebert Dec 24 '21 at 20:21