0

TL;DR: I can't get session affinity in Kubernetes work using the official nginx ingress helm chart. I've tried these instructions:

Sticky Sessions - nginx ingress controller
Using Session Affinity on Kubernetes

I've also tried to read up on the service handling. However, I get no effect whatsoever, with requests being pointed to my back end service and then load balanced by its headless service.

More information on my case: I have a public API running with three pods in Kubernetes, with an nginx ingress in front of it. The cluster is on AWS and set up using Kops.

For a third-party user, I've set up digest auth to be able to return some sensitive data. I've used a fork of this digest auth package.

All fine and dandy, works swell in dev.

However, after I deploy my system, giving it three stateless pods, the first digest request will end up on one pod, and the client's response will end up on another pod, breaking the functionality.

What is it I'm missing? All I want is that a client's IP address will always have its request ending up on a single pod in my deployment.

Thanks!

Helge Talvik Söderström
  • 161
  • 6
  • Sorry, but I have to ask this, how did you check if the Session affinity is in effect ?, you didn`t mention it in your question. – Nepomucen Apr 02 '19 at 16:04
  • Have you verified, if session affinity had propagated to your ELB configiration settings ? – Nepomucen Apr 02 '19 at 16:04
  • You don`t Set-Cookie in HTTP response header from API call ? – Nepomucen Apr 02 '19 at 16:04
  • Thanks for your replies! As far as I can troubleshoot my own system, I don't see any of the errors I would expect (or success) if the requests reach their own pods. The behaviour is consistent with what I would expect with no session affinity in place. I can't use cookie settings since the end client is an API that doesn't save cookies. I need client IP to work. I haven't checked the ELB configuration specifically, but I'll definitely start troubleshooting there! – Helge Talvik Söderström Apr 02 '19 at 16:41
  • "I can't use cookie settings since the end client is an API that doesn't save cookies". I think it does not matter here, as cookie is being set to clients by your Ingress Controller per host/path rule in Ingress resource. You should go for this option. The information in the cookie ensures that all requests are handled by the same upstream server throughout the session. Actually 'cookie' is the only affinity type available in kubernetes/ingress-nginx controller. – Nepomucen Apr 02 '19 at 17:07

1 Answers1

2

I found the answer, and whaddyaknow! It was my own fault, of course.

The problem was that I tried to configure the nginx ingress chart, which was not the way to go. The nginx ingress only provides the functionality - the services you run yourself provide the requested behaviour.

In my own service that I had deployed, I had an ingress-statement. Adding the annotation for cookies there solved my issue.

Also, creds to Nepomucen for informing me in the comments that the cookie affinity would work even though it's a bot user.

Here's my full helm chart for my own service that relied on the nginx ingress:

# Default values for chart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
image:
  repository: [redacted]
  tag: [redacted]
  pullPolicy: IfNotPresent
  imagePullSecret: [redacted] # Must be registered with the namespace.
service:
  name: api
  type: ClusterIP
  externalPort: 8080
  internalPort: 8080
  livenessProbe: /alive
  readinessProbe: /ready
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: "cookie" # THIS WAS THE REQUIRED LINE.
    #kubernetes.io/tls-acme: 'true' # Kube Lego to obtain Let's Encrypt SSL certificates
  hosts:
    - [redacted]
  tls:
    - secretName: tls
      hosts:
        - [redacted]
hpa:
  minReplicas: 3
  maxReplicas: 10
  cpuAvg: 65
resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi
env:
- name: APPLICATION_ENV
  value: "production"
- name: TZ
  value: "Europe/Stockholm"
Helge Talvik Söderström
  • 161
  • 6