I have the following (very simplified) network :
(ingress) -> DMZ (nginx) -> HA Proxy -> Reverse proxy (nginx) -> Application Server (tomcat)
All the layers send information to Prometheus
and then we use Grafana
to monitor them.
We had some latency issues. So we had to analyse each layer individually to find which layer is the slowest.
Question : By selecting a random request (from the DMZ), is there a way to trace it through the layers it went through and having the time it took at each level ? Something like :
(Request) ->
DMZ (nginx) -> 2ms
HA Proxy -> 1 ms
Reverse proxy (nginx) -> 1ms
Application Server (tomcat) 15ms
Reverse proxy (nginx) -> 1ms
HA Proxy -> 1 ms
DMZ (nginx) -> 2ms
(Response)