Unable to fetch Vault Token for Pod Service Account

Question

I am using Vault CSI Driver on Charmed Kubernetes v1.19 where I'm trying to retrieve secrets from Vault for a pod running in a separate namespace (webapp) with its own service account (webapp-sa) following the steps in the blog.

As I have been able to understand so far, the Pod is trying authenticate to the Kubernetes API, so that later it can generate a Vault token to access the secret from Vault.

$ kubectl get po webapp
NAME     READY   STATUS              RESTARTS   AGE
webapp   0/1     ContainerCreating   0          22m

It appears to me there's some issue authenticating with the Kubernetes API.

The pod remains stuck in the Container Creating state with the message - failed to create a service account token for requesting pod

Events:
  Type     Reason       Age                       From               Message
  ----     ------       ----                      ----               -------
  Normal   Scheduled    35m                       default-scheduler  Successfully assigned webapp/webapp to host-03

  Warning  FailedMount  4m38s (x23 over 35m)      kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod webapp/webapp, err: rpc error: code = Unknown desc = error making mount request: **failed to create a service account token for requesting pod** {webapp xxxx webapp webapp-sa}: the server could not find the requested resource

I can get the vault token using cli in the pod namespace:

$ vault write auth/kubernetes/login role=database jwt=$SA_JWT_TOKEN
Key                                       Value
---                                       -----
token                                     <snipped>

I do get the vault token using the API as well:

$ curl --request POST --data @payload.json https://127.0.0.1:8200/v1/auth/kubernetes/login
    {
       "request_id":"1234",
       <snipped>
       "auth":{
          "client_token":"XyZ",
          "accessor":"abc",
          "policies":[
             "default",
             "webapp-policy"
          ],
          "token_policies":[
             "default",
             "webapp-policy"
          ],
          "metadata":{
             "role":"database",
             "service_account_name":"webapp-sa",
             "service_account_namespace":"webapp",
             "service_account_secret_name":"webapp-sa-token-abcd",
             "service_account_uid":"123456"
          },
          <snipped>    
       }
    }

Reference: https://www.vaultproject.io/docs/auth/kubernetes

As per the vault documentation, I've configured Vault with the Token Reviewer SA as follows:

$ cat vault-auth-service-account.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-token-review-binding
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault

Vault is configured with JWT from the Token Reviewer SA as follows:

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="< TOKEN Reviewer service account JWT>" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" 
    kubernetes_ca_cert=@ca.crt

I have defined a Vault Role to allow the webapp-sa access to the secret:

$ vault write auth/kubernetes/role/database \
    bound_service_account_names=webapp-sa \
    bound_service_account_namespaces=webapp \
    policies=webapp-policy \
    ttl=72h
Success! Data written to: auth/kubernetes/role/database

The webapp-sa is allowed access to the secret as per the Vault Policy defined as follows:

$ vault policy write webapp-policy - <<EOF
> path "secret/data/db-pass" {
>    capabilities = ["read"]
> }
> EOF
Success! Uploaded policy: webapp-policy

Pod and it's SA is defined as follows:

$ cat webapp-sa-and-pod.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: webapp-sa
---
kind: Pod
apiVersion: v1
metadata:
  name: webapp
spec:
  serviceAccountName: webapp-sa
  containers:
  - image: registry/jweissig/app:0.0.1
    name: webapp
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          providerName: vault
          secretProviderClass: "vault-database"

• Does anyone have any clue as to why the Pod won't authenticate with the Kubernetes API?
• Do I have to enable flags on the kube-apiserver for Token Review API to work?
• Is it enabled by default on Charmed Kubernetes v1.19?

Would be grateful for any help.

Regards, Sana