UDP Server behind Gcloud load balancer: receive ok, reply fails

Question

I have a UDP server set up (on a VM) behind the Google Cloud network load balancer. The server is bound to 0.0.0.0. The UDP server can receive messages that were sent to the balancer, but replies do not go back to the client. There are no errors reported, and tcpdump is showing nothing unusual. I have made sure that all the firewall rules allow this traffic, and the load balancer is set up to forward all ports.

I am by no means a networking expert, but I suspect that something is going wrong with the UDP server in that the address on which the message is received is different to the one being used for the reply (sendto()).

I am testing all of this using Python's socketserver module in the standard library as an echo server:

import SocketServer

class MyUDPHandler(SocketServer.BaseRequestHandler):
    def handle(self):
        data = self.request[0].strip()
        socket = self.request[1]
        print "{} wrote:".format(self.client_address[0])
        print data
        socket.sendto(data.upper(), self.client_address)

if __name__ == "__main__":
    HOST, PORT = "0.0.0.0", 5029
    server = SocketServer.UDPServer((HOST, PORT), MyUDPHandler)
    server.serve_forever()

The VM only has one network interface, with local IP 10.240.x.x. If I bind the UDPServer to this local IP, then messages are not even received by this server.

Without the load balancer, everything is working normally, i.e., messages are echoed correctly back to the client.

Question: What should I do to allow my UDP server to reply to messages?

EDIT: this discussion may be relevant.

Answer 1

I had to deal with a similar issue for one of my udnergraduate projects: load balancing DNS.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Load_Balancer_Administration/s1-initial-setup-forwarding-VSA.html

/etc/sysctl.conf

 net.ipv4.ip_nonlocal_bind = 1

Then attempt to bind the python application to the public IP of the load balancer.

Basically, your server is responding on its private IP and the client is expecting a respond from the load balancer.