5

I'm having trouble exposing my Redis Cluster on Kubernetes to external applications. Using a Kubernetes load balancer service, I'm able to assign an external IP to Redis which provides initial connectivity. The trouble is, whenever the client receives a MOVED command, the IP address is a Kubernetes internal POD IP which is inaccessible by redis clients outside the cluster.

Here's an example session from redis-cli to demonstrate:

10.150.0.5:7000> set test value
-> Redirected to slot [6918] located at 10.28.1.9:6379
Could not connect to Redis at 10.28.1.9:6379: Operation timed out

How do you solve this? None of the tutorials / guides I've read shed much light on exposing the cluster to external services. Most are concerned with getting the Redis cluster setup in K8s.

shrumm
  • 116
  • 1
  • 9

3 Answers3

3

For anyone still searching for an answer, the best option I've found is to use the (now official) Redis Proxy https://github.com/RedisLabs/redis-cluster-proxy.

Note: As Yogesh mentions in his comment - this is alpha code - so please avoid doing this in production workloads till the code is stable / you know what you're doing.

  1. Setup the Redis cluster in K8's
  2. Deploy the redis-cluster-proxy and set it up to connect to your redis cluster
  3. Create a K8's external service to point to the redis-cluster-proxy instance, NOT the actual redis cluster instance
  4. Your apps should use this service to connect to Redis

Explanation The proxy basically 'pretends' to be a single Redis instance, so clients connecting to it don't need to be cluster aware.

You should use a recent version of Redis, preferably v6 or higher.

shrumm
  • 116
  • 1
  • 9
  • Though this answer is accepted, I strongly recommend not to use this in production as of today's date. Since this is in alpha phase and they are itself saying not to give try in PROD right now. – Yogesh Jilhawar Apr 01 '21 at 07:48
  • @YogeshJilhawar thanks - I've updated the answer to mention this. – shrumm Apr 26 '21 at 15:36
0

UPDATE:

I think I'm not clear about the question. And after the OP's comment. I'm going to look through the redis cluster set up and similar issues.

Here are what I found

From this Github Issue When Redis does that "Redirected to slot xxx" behavior, it doesn't proxy the connection through the Redis server, but rather simply passes the IP address of the appropriate server back to the client, which then initiates the connection directly.

From this Github Issue In a case when one of the docker containers dies, and a new container comes back up, we reattach nodes.conf from a volume to it so that It can join the currently running Redis cluster again automatically. However, when the container comes back, it comes with a different IP address. And when we start new Redis server inside this container it does not update the IP address inside nodes.conf file for the only new generated container. But all other nodes have awareness of new IP address inside their nodes.conf. Overall, a node which went down - does not update its own IP address inside nodes.conf

So the conclusion here is we need to update the IP of the pod to the nodes.conf file after restarting, so it can join cluster. The example can be found here by adding script to add podIP to nodes.conf after restarting.

OLD ANSWER:

If you are exposing using LoadBalancer service, consider using static IP so that it won't be allocated to new IP.

Here is a sample

apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379
  type: LoadBalancer
  loadBalancerIP: "YOUR.IP.ADDRESS.HERE"

Replace YOUR.IP.ADDRESS.HERE with a reserved internal IP (and remember that it is a regional IP).

How to reserve internal IP in GCP

Tranvu Xuannhat
  • 146
  • 2
  • Thanks, the issue is not that the IP is dynamic. Let me know if the problem statement is not clear - not sure if you're familiar with how Redis cluster works? The server issues a `MOVED` command with an internal IP. So it doesn't matter what you configure on the service, the MOVED command ignores it. – shrumm Sep 30 '19 at 12:20
  • Hi. I take a bit time to read through the redis cluster set up and look for some similar issues. I update my answer with my findings. Hope this can help you. If it cannot help, we can discuss more. I'm also interested in redis cluster in GKE. – Tranvu Xuannhat Oct 01 '19 at 07:55
  • Sorry Tranvu =) I really appreciate the time you've taken, but this is not the answer and it's pretty clear you're unfamiliar with how Redis Cluster works. Thanks again for your time. Since you mention you're interested in these topics, I'd encourage you to try it - install Redis Cluster on K8's and try to expose it to applicaitons outside the cluster - you will see what I mean. – shrumm Oct 01 '19 at 08:57
0

I have researched this issue on the Google Cloud Platform side, as it is my area of expertise, and have discovered the following documentation[1]. Other documentation regarding Redis [2] has confirmed your assumptions of how Redis handles IPs. It appears as though Redis nodes will use their own IP when sending to the client, thusly, when the client replies (through a load balancer) the wrong IP is being used. This behaviour might be intentional by the Redis designers. I have not been able to find a work-around, and I must admit it is out of scope for me. Some food for thought: although GKE can be used with a database it is not best practice. If you do want to use a database in k8s, you should use a StatefulSet which will try to maintain the same pod ip. Google does offer alternate methods of integrating into the Cloud. There are several products available , maybe some time spent on reviewing your needs and the capabilities of Google products would be fruitful. One in particular would be Cloud Memorystore as it’s intended use is with Redis. The documentation of this product states : “ Cloud Memorystore for Redis is fully Redis protocol compliant.” [3]

[1] Connecting to a Redis instance: https://cloud.google.com/memorystore/docs/redis/connect-redis-instance-gke [2] Client and Server roles : https://redis.io/topics/cluster-spec [3] Overview of Cloud Memorystore for Redis : https://cloud.google.com/memorystore/docs/redis/redis-overview

Derek Fender
  • 41
  • 2