I've recently upgraded my on-premises Kubernetes cluster from 1.12.1 to 1.13.0 for obvious reasons, but now my kubectl running on the master node cannot get the container logs anymore from any node, except the master itself.
This issue happens on both the nodes where kubelet has been upgraded to 1.13.0, but also on the nodes where kubelet is still running under version 1.12.1. This makes me think that it is not the problem with kubelet itself, but something in the configuration handling in version 1.13.0 changed that causes the logging to fail.
When I do a kubectl logs -f pod/podname-749c5c589f-vt7hj
, I get:
Error from server: Get https://10.48.76.46:10250/containerLogs/default/podname-749c5c589f-vt7hj/podname?follow=true: Forbidden
I've checked all the obvious things like network connections between the master and the minion running this pod, as any proxies that might be configured, but it keeps failing. Also the kubelet log on the minion node doesn't log any error.
Furthermore, other usual kubelet commands for this pod, like attach
and exec
work fine for this pod, so the inter-node communication is working fine. Also the pod itself is running fine, I just cannot see its logs.
Any idea what happened in 1.13.0 that might cause this to break?