0

I am using a pod (image bitnami/kubectl) in a kubernetes cluster (clusterA) but kubeconfig is set to point to another cluster (clusterB).

Using on premises kubernetes 1.21.7 (on VMs) installed via kubeadm.

kubeconfig used:

apiVersion: v1
kind: Config
clusters:
- name: default-cluster
  cluster:
    certificate-authority-data: XXXXXXXXXXXXXXXXXXXXXX
    server: https://clusterB:6443
contexts:
- name: default-context
  context:
    cluster: default-cluster
    namespace: default
    user: default-user
current-context: default-context
users:
- name: default-user
  user:
    token: YYYYYYYYYYYYYYYYYYYYYY

But even though kubeconfig is correct, clusterA doesn't allow the request to leave clusterA to reach clusterB. It seems kubernetes control plane in clusterA interprets the pod kubectl request as if it is trying to control clusterA itself but that's not what I am trying to do. I am trying to reach clusterB instead (defined in 'server' in kubeconfig).

This is the error when kubectl runs:

Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=secrets", GroupVersionKind: "/v1, Kind=Secret"
Name: "mysecret", Namespace: "istio-system"
from server for: "STDIN": secrets "mysecret" is forbidden: User "system:serviceaccount:mynamespace:default" cannot get resource "secrets" in API group "" in the namespace "istio-system"

EDIT: I am using kubectl setting a custom kubeconfig:

# kubectl --kubeconfig=/etc/custom.kubeconfig cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Error from server (Forbidden): services is forbidden: User "system:serviceaccount:mynamespace:default" cannot list resource "services" in API group "" in the namespace "kube-system"

# grep server /etc/letsencrypt/custom.kubeconfig
server: https://clusterB:6443

EDIT2:

# kubectl --kubeconfig=/etc/custom.kubeconfig cluster-info dump
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:cadeado--producao:default" cannot list resource "nodes" in API group "" at the cluster scope
brgsousa
  • 215
  • 1
  • 4
  • 11
  • Which version of Kubernetes did you use and how did you set up the cluster? Did you use bare metal installation or some cloud provider? It is important to reproduce your problem. – Mikołaj Głodziak Mar 23 '22 at 12:21
  • Using on premises kubernetes 1.21.7 (on VMs) installed via kubeadm. – brgsousa Mar 23 '22 at 12:49
  • Did you try to run this command: `kubectl cluster-info dump`? Could you paste the results to the question? – Mikołaj Głodziak Mar 25 '22 at 13:05
  • I added a second edit – brgsousa Mar 30 '22 at 19:45
  • Please see [this similar problem](https://stackoverflow.com/questions/69535118/user-systemserviceaccountdefaultflink-cannot-list-resource-nodes-in-api-g). It is related to another product, but in the answer you will get information what should you check and how can you repair your problem. Please let me know if it was helpful. – Mikołaj Głodziak Mar 31 '22 at 07:44
  • Kind of. The issue in that post is inside the same cluster. What I am trying to do here is a "cross-cluster" access. Pod in cluster A is trying to access cluster B – brgsousa Mar 31 '22 at 18:50
  • I have one more [similar problem](https://stackoverflow.com/questions/67151953/forbidden-resource-in-api-group-at-the-cluster-scope) to yours. If it will be not helpful please run the command `kubectl auth can-i list nodes ` (similar to the command from linked question) and paste here results. – Mikołaj Głodziak Apr 01 '22 at 09:42

1 Answers1

0

After issuing following from pod:

kubectl cluster-info

what do you get?

Seems the problem you have is related to user access, you need to provide
default-user with appropriate RBAC rights.

mforsetti
  • 2,488
  • 2
  • 14
  • 20
gayan ranasinghe
  • 11
  • 1
  • Got this error:To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Error from server (Forbidden): services is forbidden: User "system:serviceaccount:mynamespace:default" cannot list resource "services" in API group "" in the namespace "kube-system" – brgsousa Mar 23 '22 at 12:50