Extending chrskly's answer, you might want to configure 3 flags/configs.
- fail_timeout : Total time by failed attempts and also mark the server as DOWN for that same time. If 5 sec, then will try max_fail attempts in 5 secs and if still fails, mark that server as DOWN for 5 sec.
- max_fail : Maximum number of tries
- proxy_connect_timeout : The amount of time to wait for a connection.
In following GRPC example, if main server can't be connected in 7 sec, then switch to backup and mark main server as down for 6000s:
upstream grpcservers {
server 192.168.0.XX:9997 fail_timeout=6000s max_fails=1; # After 1 fail in 6000s, Main server is marked unavailable for 6000s.
server 192.168.0.XX:9999 backup;
}
location / {
grpc_pass grpc://grpcservers;
grpc_connect_timeout 7s; # If conn cant be made in 7sec, switch to backup
}