I have a local kubernetes cluster using kind. It is a single node cluster.
On this cluster I am following this guide to setup Vault & the vault-agent-injector.
If I follow the tutorial step by step the end result is that the orgchart pod will fail to start as it is still waiting for the vault-agent sidecar to start.
kubectl get pods:
NAMESPACE NAME READY STATUS RESTARTS AGE
default dnstools 0/1 Completed 0 22m
default orgchart-7457f8489d-hxtqt 1/1 Running 0 26m
default orgchart-798cbc6c76-4kzrl 0/2 Init:0/1 0 25m
default vault-0 1/1 Running 0 28m
default vault-agent-injector-79d479cf7d-kz67q 1/1 Running 0 28m
kube-system coredns-558bd4d5db-gnxl7 1/1 Running 0 45m
kube-system coredns-558bd4d5db-ph95m 1/1 Running 0 45m
kube-system etcd-kind-control-plane 1/1 Running 0 45m
kube-system kindnet-bjvzg 1/1 Running 0 45m
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 45m
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 45m
kube-system kube-proxy-v5545 1/1 Running 0 45m
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 45m
local-path-storage local-path-provisioner-547f784dff-mp79j 1/1 Running 0 45m
kubectl get svc:
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 44m
default vault ClusterIP 10.96.42.180 <none> 8200/TCP,8201/TCP 27m
default vault-agent-injector-svc ClusterIP 10.96.156.79 <none> 443/TCP 27m
default vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 27m
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 44m
kubectl describe org:
Name: orgchart-798cbc6c76-4kzrl
Namespace: default
Priority: 0
Node: kind-control-plane/172.18.0.2
Start Time: Fri, 20 Aug 2021 13:07:13 +0000
Labels: app=orgchart
pod-template-hash=798cbc6c76
Annotations: vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-secret-database-config.txt: internal/data/database/config
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/role: internal-app
Status: Pending
IP: 10.244.0.12
IPs:
IP: 10.244.0.12
Controlled By: ReplicaSet/orgchart-798cbc6c76
Init Containers:
vault-agent-init:
Container ID: containerd://d7dc0c9d089a1e59d85ac16a422a5195f5e7026a5c98a004d5e6a6a576900119
Image: hashicorp/vault:1.8.0
Image ID: docker.io/hashicorp/vault@sha256:54d12f84c0bfbed57d2d2e1d89099540c5ccb4e6c3b2b0a123b82dab12c99e75
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Running
Started: Fri, 20 Aug 2021 13:07:14 +0000
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: <REDACTED>
Mounts:
/home/vault from home-init (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx52m (ro)
/vault/secrets from vault-secrets (rw)
Containers:
orgchart:
Container ID:
Image: jweissig/app:0.0.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx52m (ro)
/vault/secrets from vault-secrets (rw)
vault-agent:
Container ID:
Image: hashicorp/vault:1.8.0
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: <REDACTED>
Mounts:
/home/vault from home-sidecar (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lx52m (ro)
/vault/secrets from vault-secrets (rw)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-api-access-lx52m:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
home-init:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
home-sidecar:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
vault-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 26m default-scheduler Successfully assigned default/orgchart-798cbc6c76-4kzrl to kind-control-plane
Normal Pulled 26m kubelet Container image "hashicorp/vault:1.8.0" already present on machine
Normal Created 26m kubelet Created container vault-agent-init
Normal Started 26m kubelet Started container vault-agent-init
Warning DNSConfigForming 53s (x25 over 26m) kubelet Search Line limits were exceeded, some search paths have been omitted, the applied search line is: default.svc.cluster.local svc.cluster.local cluster.local <REDACTED> <REDACTED> <REDACTED>
Looking in the apiserver logs I see the following message:
E0820 13:03:55.315142 1 dispatcher.go:171] failed calling webhook "vault.hashicorp.com": Post "https://vault-agent-injector-svc.default.svc:443/mutate?timeout=10s": dial tcp 10.96.156.79:443: connect: no route to host
And looking in the vault-agent-injector logs I see the following message:
2021-08-20T13:03:55.887Z [INFO] handler: Starting handler..
Listening on ":8080"...
2021-08-20T13:03:55.896Z [INFO] handler.auto-tls: Generated CA
2021-08-20T13:03:55.899Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-08-20T13:05:55.808Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-08-20T13:07:13.447Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-08-20T13:10:16.613Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
I have configured my kind cluster to be able use MutatingAdmissionWebhooks:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
- |
kind: ClusterConfiguration
apiServer:
extraArgs:
enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
extraPortMappings:
- containerPort: 80
hostPort: 8080
protocol: TCP
- containerPort: 443
hostPort: 8443
protocol: TCP
networking:
apiServerAddress: "127.0.0.1"
apiServerPort: 443
What next steps can I take to help me debug the issue?
I am thinking this is an issue with the DNS lookup for the vault-agent-injector-svc but am not sure where to start.
