2

I have a Job that I'd like to retry a few times in Kubernetes. A Job can have a backoffLimit and a activeDeadlineSeconds set. Cool so far.

However, the latter setting takes precedence over the first, unfortunately. My Job blocks for a long time until it fails, so I'd like to have a way to retry the task in a new pod with each retry having a deadline.

E.g. try five times with a deadline of 30 seconds for each attempt.

Is that possible in Kubernetes?

gertvdijk
  • 3,362
  • 4
  • 30
  • 46

1 Answers1

3

How can I set a deadline/timeout for each try (backoffLimit) in a Kubernetes Job?

...

Is that possible in Kubernetes?

You cannot do that in K8s, you will have to implement a logic into the script or image that you are using inside the pod.

As we can read in the documentation Job Termination and Cleanup which was mentioned by You.

Note that a Job’s .spec.activeDeadlineSeconds takes precedence over its .spec.backoffLimit. Therefore, a Job that is retrying one or more failed Pods will not deploy additional Pods once it reaches the time limit specified by activeDeadlineSeconds, even if the backoffLimit is not yet reached.

Community
  • 1
Crou
  • 714
  • 3
  • 9
  • 1
    Thanks for the confirmation. It's a little bit sad to see this, especially given the ecosystem of k8s pushing you to do 'retries' instead of specifying dependencies, but does not allow you to retry this properly. Anyway, I've used GNU's `timeout` to wrap the container's entrypoint in, which does the job for now (pun intended). `:-)` – gertvdijk Jul 22 '19 at 10:26