I am trying to use IAM roles for service accounts in EKS. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
When it comes to create the IAM role to be assigned to a service account, I have to create it with a trust policy that refers to the OIDC provider of a specific cluster.
https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html
Specifically:
You must also create an IAM role for your Kubernetes service accounts to use before you associate it with a service account. The trust relationship is scoped to your cluster and service account so that each cluster and service account combination requires its own role
This part: The trust relationship is scoped to your cluster and service account is REALLY limiting.
My issue with this is that I would like clusters to be ephemeral and short lived, but I don't want to touch up the trust policy of all the application IAM roles assigned to Service accounts in the cluster every time I rebuild a cluster. Furthermore, I would like the same kubernetes manifest manifest (for the same service account) to be applied, identical, to more than one cluster, potentially on clusters that didn't exist when the manifest was written and the application IAM role was created.
In pre-eks time, when using kube2iam, I was simply creating cluster nodes that shared the instance-profile, and use the role of the instance profile in the trust policy of the IAM roles assigned to pods. That allowed me to write manifests that would work on more than one cluster.
I cannot, apparently, share an OIDC provider between two EKS clusters, in a similar way as I was sharing the node role (instance-profile) before.
Am I approaching the problem from the wrong end? I don't want to hardcode in the trust policy of the application IAM roles something (in this case the OIDC provider) that is cluster specific, because the cluster should not be eternal. I want to be able to rebuild it from scratch without rebuilding the roles of the applications (or the service account manifests) that I might run on it.
I would stick to kube2iam, but that doesn't work with fargate profiles, so IAM roles for service accounts is my only solution.
