Consume kafka topic behind a NAT

Question

We have a kafka cluster with 3 broker and 3 zookeeper that work fine. We want to consume the topic from a consumer that can only access the kafka broker behind natted address. Can you share any config examples on how make it work? Right now, even if the flow are working, the consumer can connect but then it seems to be confused about the ip/hosts that result with some io timeout, and the ip given are the one on which the kafka is listening to and not the natted addresses of course. I have read this how-to: https://rmoff.net/2018/08/02/kafka-listeners-explained/ but not really sure if the different scenario apple to me setup... Thanks!

Answer 1

Ok I think I have found the issue. It's important to not use the ip addresses in the server.properties for: advertised.listeners

let says that you have:

listeners=INTERNAL://<ip>:19092,EXTERNAL://<ip>:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://<hostname>:19092,EXTERNAL://<hostname>:9092
inter.broker.listener.name=INTERNAL

should be resolvable from your broker server point of view but also from your consumer point of view. So, if resolve to in your broker server, on your consumer side, if it is behind a nat, you add and hosts entry in your /etc/hosts file or your dns server so that will point to the nap ip of your broker server. Because the consumer get the list from your broker cluster, it need to contact them. By using name instead of ip, you can make sure it can find them wherever they are.