vault init hangs on kubernetes

Question

I'm trying to set up an autosealing vault cluster in kubernetes but I'm seeing some strange behaviour.

I have one vault providing the transit secret to autounseal the second vault . They are running in the same k8s cluster in separate namespaces. The second vault runs within a pod with an auto start script (see below) but when it runs vault init hangs and eventually returns a 2 code (timeout) despite the vault instance is successfully initialized and unsealed.

The problem is that I'm trying to init the second vault with a post-start script in its pod and the error code 2 breaks the pod.

Has anyone seen similar behaviour or can help solving it?

apiVersion: v1
kind: ConfigMap
metadata:
  name: post-start
data:
   post-start.sh: |
#!/bin/sh

#redirect stdout and stderr to kube logs
# exec &> /proc/1/fd/1

export VAULT_CLIENT_TIMEOUT=240

echo $VAULT_CLIENT_TIMEOUT > /proc/1/fd/1

nc -z 127.0.0.1 8200
while [ $? = 1 ]; do
  sleep 2
  nc -z 127.0.0.1 8200
done

echo "port 8200 ready" > /proc/1/fd/1
vault init

aga · Answer 1 · 2019-08-21T06:41:50.357

0

Please refer to documentation of defining postStart and preStop handlers and for container lifecycle hooks:

Hook delivery is intended to be at least once, which means that a hook may be called multiple times for any given event, such as for PostStart or PreStop. It is up to the hook implementation to handle this correctly.

Look for example:

  lifecycle:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            if [ -s /var/www/mybb/inc/config.php ]; then
            rm -rf /var/www/mybb/install;
            fi;
            if [ ! -f /var/www/mybb/index.php ]; then
            cp -rp /originroot/var/www/mybb/. /var/www/mybb/;
            fi

Also, please provide logs from crashed POD.

edited Aug 21 '19 at 06:41

answered Aug 16 '19 at 12:33

aga

128
3

Thanks for your response. I think the problem is in vault as the script runs successfully on pod boot. The main problem is that it looks like vault accepts the request to init but somehow doesn't report back to the script. Anyway, I'll try reproduce the issue and post the logs here. – Javier PR Aug 21 '19 at 06:29
After reading the pages you pointed out, I reckon that this might be the key: `Hook delivery is intended to be at least once, which means that a hook may be called multiple times for any given event, such as for PostStart or PreStop. It is up to the hook implementation to handle this correctly.` – Javier PR Aug 21 '19 at 06:34
Thanks for yor comments they were useful to better understand who post-start works but the problem was in vault not in the script itself. – Javier PR Aug 22 '19 at 07:11

score 0 · Accepted Answer · answered Aug 22 '19 at 07:10

After some fiddling it turns out that the real problem was that port 8200 was never ready because the second vault didn't start. Second vault didn't start because it was unable to communicate with vault 1 and I sorted that out setting TLS properly (this link might come in handy)

After configuring TLS, vault 2 was able to start, and the post-script finished succesfully.

vault init hangs on kubernetes

2 Answers2