Stackdriver missing GKE Logs

Question

I brought up a new Vault cluster using Terraform (https://github.com/sethvargo/vault-on-gke) and everything is working fine with Vault. However I can't seem to see any Container, Node or Pod logs in Stackdriver. The other GKE Cluster we have that was brought up manually is working fine.

I can view the logs on the pods using kubectl logs without a problem.

Existing cluster that works fine

Terraformed vault cluster

I'm completely stumped as to why it's not working. Stackdriver is enabled for the cluster, and the beta Stackdriver box is ticked - same as the working cluster. The Compute service account has Editor role, same in both account.

Any thoughts?

Phylu · Answer 1 · 2018-12-29T15:57:13.270

4

You need to make sure that your kubernetes cluster has the correct permissions for logging and monitoring.

To have a properly running cluster, the kubernetes master needs the following oauth scopes (the latter two are for stackdriver):

compute-rw (https://www.googleapis.com/auth/compute)
storage-ro (https://www.googleapis.com/auth/devstorage.read_only)
logging-write (https://www.googleapis.com/auth/logging.write)
monitoring (https://www.googleapis.com/auth/monitoring)

In addition, the service account of the nodes in the kubernetes cluster need the following permissions:

roles/logging.logWriter (https://cloud.google.com/iam/docs/understanding-roles#stackdriver_logging_roles)
roles/monitoring.metricWriter (https://cloud.google.com/iam/docs/understanding-roles#stackdriver_monitoring_roles)

For more information see:

edited Dec 29 '18 at 15:57

answered Nov 30 '18 at 15:48

Phylu

41
3

Welcome to serverfault. When the link breaks, your answer becomes worthless later even if the linked material answered the question. Please include at least a summary so the answer can stand on its own. Thank you! – marsh-wiggle Nov 30 '18 at 16:07
1

Thanks. In my case, I configured a custom service account for the GCE instances and it lacked permissions. – Jonathan Lin Dec 28 '18 at 10:33
The part regarding missing roles for service account fixed it for me! I added the log writer and metric writer roles to my default service account for Compute Engine and logs magically started to appear in StackDriver. – Cécile Fecherolle May 19 '21 at 08:26

Patrick W · Answer 2 · 2018-11-08T20:10:45.080

0

If you are looking for the equivalent of kubectl logs, you should be looking in 'kubernetes pod' or 'kubernetes container' for the logs. 'kubernetes cluster' shows you cluster activity such as resource patches and updates.

The next thing to check is to make sure that logging shows 'Enabled v2(beta)' and that the stackdriver agent is properly running in your cluster using 'kubectl get po -n kube-system | grep fluentd'

If all the above checks out, I recommend creating a public issue tracker as there is likely something wrong with either the cluster version of the beta logging

edited Nov 08 '18 at 20:10

answered Nov 07 '18 at 16:31

Patrick W

582
2
8

That's exactly my problem. I don't have Kubernetes pod or Kubernetes container showing up in Stackdriver for this cluster. Look at the screen shot in the link Terraformed vault cluster. – Max DiOrio Nov 08 '18 at 17:39
Sorry about that Max, I've added more steps. This looks like a bug to me – Patrick W Nov 08 '18 at 20:11
Beta is enabled and fluentd is running on all nodes. Thanks for the info. – Max DiOrio Nov 09 '18 at 21:10

Stackdriver missing GKE Logs

2 Answers2