We have a Kubernetes CronJob resource that runs a Job every minute. I need the container that results from the Job to know when it was scheduled to run. Is this possible?
Background info just in case it's useful
The hierarchy of Kubernetes resources is this: CronJob -> Job -> Pod -> initContainer -> Container -> PHP CLI command.
I have noticed the container will start between 20 seconds and 10 minutes after the minute which we wanted it to run. There are 2 reasons for the significant startup time:
- The
init-container
does quite a few things including pulling docker images, and it also does acp
of the >300MB app sourecode into a volume. - The
container
resource limits are rather high, e.g 2G memory, meaning there sometimes isn't enough capacity in the cluster so the cluster-autoscaler has to provision a new node to run the Job Pod, and the bootstrapping process can take a while before the new node joins the cluster.
This start up delay can have interesting effects such as a Job scheduled later can beat a Job scheduled earlier to run because it happens to have the capacity it needs to run whereas the previous one didn't so is waiting for a new node to start.
Some things I've looked at so far to solve my problem
- I looked at the DownWards API so the pod can look in
/etc/labels
to know what labels it has but unfortunately it does not provide pod startup time, only pod name. - I looked at using a dynamic value in a Pod label, i.e. the current timestamp. But as far as I know, this isn't possible?
Versions
Kubernetes v1.10, running in AWS EKS. Job is a PHP 7.2 CLI command.
Steps to reproduce
- Set up a Kubernetes cluster with a cluster-autoscaler installed and enabled.
- Create a
CronJob
. Put the following yaml in a file calledcron_test.yaml
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: tomtest
labels:
app: test
tier: test
tester: tom
spec:
schedule: "* * * * *"
jobTemplate:
metadata:
name: tomtest-crons
spec:
template:
metadata:
labels:
app: test
tier: test
tester: tom
build_id: tom3
spec:
containers:
- name: cron
image: giantswarm/tiny-tools
imagePullPolicy: IfNotPresent
env:
- name: TOMTEST
value: "3"
args:
- /bin/sh
- -c
- date;echo hi;sleep 600;echo bye;date
restartPolicy: Never
resources:
requests:
cpu: "1"
memory: "2G"
limits:
cpu: "1"
memory: "2G"
- Start the CronJob on your cluster:
kubectl create -f cron_test.yaml
- This will launch a container each minute which does nothing but sleep for 10 minutes.
- Wait a few minutes, the containers will start to stack up and because they have high resource limits it's likely the cluster-autoscaler will kick in and add a new node or 2. If not, increase the resource limits further.
- Do
kubectl get pods
to find Pods that seem late - i.e. that have a startup time that isn't exactly a minute from the previous one. - Inspect the Pod info:
kubectl get pod tomtest-123-456 -o=yaml
- notice there is acreationTimestamp
field and astartTime
but these are not when the Pod was actually scheduled. - When finished, clean up:
kubectl delete CronJob tomtest
(This removes all Jobs and Pods as well)