0

I'm trying to debug why my application logs aren't being picked up by Promtail for ingestion into Loki.

In order to exclude the complexities of my PHP application logging being the culprit, I'm simply trying to print a message to the stderr stream from the pod's shell to be picked up by kubectl logs counter. In order to reduce the complexity further I've spun up a brand new pod using the php:fpm-alpine image using the below config:

apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
    - name: count
      image: php:fpm-alpine

Once the pod is ready, I access the shell by running kubectl exec -it counter -- /bin/ash and then run i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done. I can see the following output in the shell of the pod:

0: Sun May 24 13:51:09 UTC 2020
1: Sun May 24 13:51:10 UTC 2020
2: Sun May 24 13:51:11 UTC 2020
3: Sun May 24 13:51:12 UTC 2020
4: Sun May 24 13:51:13 UTC 2020

etc...

However when I run kubectl logs counter the data returned is:

[24-May-2020 13:36:20] NOTICE: fpm is running, pid 1
[24-May-2020 13:36:20] NOTICE: ready to handle connections

I can't seem to get the output of the i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done command to show in the pod logs.

Alex Godbehere
  • 101
  • 4

1 Answers1

2

It appears to be due to that image enabling catch_workers_output = yes

In the future, it would be a great troubleshooting mechanism to try and run that same image locally, and see if you can create a minimum reproduction for yourself: that is, can any php script run under that container emit output to the container's stderr?

mdaniel
  • 2,338
  • 1
  • 8
  • 13
  • Good spot, thanks. What would be the best way to change this in my image? With this setting active as it currently is, how are the `[24-May-2020 13:36:20] NOTICE: fpm is running, pid 1` logs getting through? – Alex Godbehere May 24 '20 at 19:53
  • Regarding the second question, after digging deeper I'm assuming it's to do with the `error_log = /proc/self/fd/2` line? – Alex Godbehere May 24 '20 at 20:02