0

My problem occured when using Redis on Kubernetes, but it seems that it is not a problem with Redis itself, but with network/infrastructure. My scenario:

  • I have a Redis Service with single Redis Pod serving it.
  • I connect Redis Client to the Service.
  • I delete a Redis Pod.
  • Client connection gets ended.
  • Redis Client tries to reconnect.
  • In this time Redis Replica Ret brings up a new Redis Pod, and the Redis Service starts responding to requests/creating new connections.
  • However my existing Redis Client is hanged on connection (first reconnect attempt) and it stays that way until it gets timeout (which is approximately after 130 seconds).
  • On the second reconnect attepmt it gets connected immediately.

The problem seems to not exists on my dev env (local docker containers), because timeout shows up after a second or 2.

Also, the client that I am using has no option to configure a socket timeout.

  1. Is this a proper behavior of a Service - hanging a connection until a timeout occurs whend there are no Pods to handle requests? If it responded with error immediately, there would be no such problem.
  2. Is there a way to configure this timeout to acceptable value somewhere (on Service level, on some network configuration, etc.)? Let's say 5 seconds would be ok.
rideronthestorm
  • 101
  • 1
  • 2
  • you need to check a behavior of a new Redis Pod. Try to look through its logs using `kubectl log ` command and understand what amount of time is required to start the application – Artem Golenyaev Jun 26 '18 at 15:51
  • It's not a pod or redis server problem. New redis pod starts responding in about 10-15 seconds and that's fine. Problem is despite new pod is already responding and k8s service is able to accept new connections, the connections started before there was a new pod are hanged until timeout of 130s ius reached. – rideronthestorm Jun 27 '18 at 07:58

1 Answers1

0

A Service is an abstraction in Kubernetes which defines a logical set of Pods and a policy by which to access them. Technically, a Service is a type of Kubernetes resource that causes a proxy to be configured to forward requests to a set of pods.

In your case, your Redis Client connects directly to your Redis Pod using some firewall rules configured in a Service. Therefore, if the Service was untouched, connection problems may appear only on the Pod side or on the Client side. So, you need to look for configuration settings for them.

Artem Golenyaev
  • 253
  • 1
  • 7
  • According to these issues on k8s repo, it seems that you are wrong: https://github.com/kubernetes/kubernetes/issues/47340 https://github.com/kubernetes/kubernetes/issues/48719 People reported similar problem and from these issues I conclude that it is connected with a service/kube-proxy iptables configuration in Kubernetes. – rideronthestorm Jun 27 '18 at 14:08