2

I deployed an Elasticsearch cluster on Google Kubernetes Engine. I am able to view logs of specific pods with the 'kubectl logs' command from my PC.

>kubectl logs es-data-0
...
[2018-06-18T07:35:11,220][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-0] [gc][4510405] overhead, spent [290ms] collecting in the last [1s]
[2018-06-18T09:48:19,194][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-0] [gc][4518376] overhead, spent [366ms] collecting in the last [1.1s]
[2018-06-18T10:01:49,344][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-0] [gc][4519183] overhead, spent [641ms] collecting in the last [1s]

I followed this tutorial to deploy a ConfigMap & DaemonSet for Stackdriver logging: https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/

the fluentd pods are running:

>kubectl get pods NAME READY STATUS RESTARTS AGE cerebro-59648dc47c-vr964 1/1 Running 0 25d es-client-7bff44b8f5-2wqcs 1/1 Running 0 12d es-client-7bff44b8f5-vnrhg 1/1 Running 0 12d es-data-0 1/1 Running 0 52d es-data-1 1/1 Running 0 52d es-data-2 1/1 Running 0 52d es-master-6bf767f949-8fpjl 1/1 Running 0 52d es-master-6bf767f949-brjpq 1/1 Running 0 52d es-master-6bf767f949-gx2jp 1/1 Running 0 52d fluentd-gcp-v2.0-7mncl 1/1 Running 0 43m fluentd-gcp-v2.0-rsfmc 1/1 Running 0 43m fluentd-gcp-v2.0-tbh9t 1/1 Running 0 43m kibana-595858b4b7-5npcr 1/1 Running 0 52d nginx-ingress-controller-86c8447687-z4rjq 1/1 Running 2 52d nginx-ingress-default-backend-6664bc64c9-q2hnm 1/1 Running 338 52d >kubectl get ds --all-namespaces NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE default fluentd-gcp-v2.0 3 3 3 3 3 beta.kubernetes.io/fluentd-ds-ready=true 44m kube-system fluentd-gcp-v2.0.10 3 3 3 3 3 beta.kubernetes.io/fluentd-ds-ready=true 52d kube-system metadata-proxy-v0.1 0 0 0 0 0 beta.kubernetes.io/metadata-proxy-ready=true 52d kube-system nvidia-gpu-device-plugin 0 0 0 0 0 <none> 52d

The ConfigMap is present:

>kubectl get cm
NAME                              DATA      AGE
cerebro-configmap                 1         52d
curator-config                    2         6d
fluentd-gcp-config                4         47m
ingress-controller-leader-nginx   0         52d
nginx-ingress-controller          1         52d

Finally, I proceeded to Logs Viewer Page, where I selected "GKE Container": https://console.cloud.google.com/logs/viewer

Unfortunately, I do not see any logs from any pods. As mentioned previously, when using "kubcetl logs" command I can view logs from all the pods from my PC.

syst0m
  • 131
  • 1
  • 4

1 Answers1

1
  • Need to make sure the cluster is set up with Stackdriver logging/monitoring enabled.
  • Following the mentioned documentation, you should have a pod named counter. If you run:

$ kubectl logs counter # Are you able to see the output for that pod?

  • If you intend to customize Stackdriver Logs for Kubernetes Engine with Fluentd, this tutorial suggests to first create the cluster service account and assign required roles. Next, the cluster is created with "no-enable-cloud-logging"; and, deploying the Fluentd daemonset.

  • Trying the advanced filter like below should help:

resource.type="container"

resource.labels.pod_id="es-data-0"

Asif
  • 41
  • 3