3

In order to make our Kafka cluster available both from Internet and from our private network, we thought fine to configure Kafka this way :

Private VIP:9000 => All brokers:9092 (topology query only)
Private VIP:9001 => Broker #1:9092
Private VIP:9002 => Broker #2:9092
...
Public VIP:9000 => All brokers:9092 (topology query only)
Public VIP:9001 => Broker #1:9092
Public VIP:9002 => Broker #2:9092
...

We configured our Load balancer this way, and then our brokers :

listeners=PLAINTEXT://<server_priv_ip>:9092
advertised.listeners=INTERNAL://<private_VIP>:9001,EXTERNAL://<public_vip>:9001
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
inter.broker.listener.name=INTERNAL

Now, as you would expect, Kafka brokers don't start :

ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: INTERNAL://<private_VIP>:9001,EXTERNAL://<public_vip>:9001

From my Administrator point of view, my approach was totally logical, although I expected a problem where producer/consumers would get both addresses wherever they contacted the broker from. Using the same port on 2 different addresses seems logical and promotes clarity...

First question : Why is this wrong ?
Second question : How can I achieve my goal if not this way? (most options are on the table)

mveroone
  • 447
  • 7
  • 22

2 Answers2

2

We have talked to a Kafka expert and here is what came out of it.

Simply put, while kafka has the Listener Names in its knowledge to differentiate 2 listeners, he needs to know which one the client that connects wants to reach and can only do so using the incoming port.

Also listeners and advertised listeners are mapped to one another if they have the same port so you need them to match.

In the end, here is what we did :

listeners=INTERNAL://hostname:900N,EXTERNAL://hostname:910N,REPLICATION:hostname:9092
advertised.listeners==INTERNAL://vip:900N,EXTERNAL://vip:910N,REPLICATION:hostname:9092  
listeners.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,REPLICATION:PLAINTEXT
inter.listener.protocol=REPLICATION

Where "N" is the broker ID (counting from 1 forwards)

On the Load balancer, we map each port of the VIP to the broker's IP address plus a Virtual IP on port 9000(internal network) and 9100 (external network) which map to the relevant listener of each broker.

That's a bit overkill, but it works as expected (as long as kafka internal metadata topics like __consumer_offset are replicated)

mveroone
  • 447
  • 7
  • 22
0

There is one scenario that is not achieveable in kafka.

  1. There is a setup of three kafka servers in cluster. Kafka Servers has private IPs and is only accessible from its own VPN say KAFKA VPN.
  2. There is one public IP that is accessible when machine is connected to KAFKA VPN + Office Netowork. This public IP is only used to NAT the private IPs so that over the internet people can access it from Office Network.
  3. There is one consumer at Office Network. Now that consumer can consume topics from Kafka using public IP when it is connected to KAFKA VPN.

Now I want to connect Public IP by disconnecting KAFKA VPN. Is that possible ? Network is working fine. I have checked Trace Route and Reversed Trace Route.

I think there is some problem with producer / kafka server / consumer configuration

NAT Configuration:: 10.XX.XX.XX:9092 -> AA.XX.XX.XX:9095 10.XX.XX.XY:9092 -> AA.XX.XX.XX:9093 10.XX.XX.XZ:9092 -> AA.XX.XX.XX:9094

Producer Configuration:: bootstrap-servers=10.XX.XX.XX:9092,10.XX.XX.XY:9092,10.XX.XX.XZ:9092

Kafka Configuration:: listeners=SASL_SSL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9095 advertised.listeners=SASL_SSL://10.XX.XX.XX:9092,EXTERNAL://AA.XX.XX.XX:9095 listener.security.protocol.map=SASL_SSL:SASL_SSL,EXTERNAL:SASL_SSL

Consumer Configuration:: sh kafka-console-consumer.sh --bootstrap-server AA.XX.XX.XX:9093,AA.XX.XX.XX:9094,AA.XX.XX.XX:9095 --topic test --consumer.config consumer.properties

Error that I am getting when I am not connected to KAFKA VPN:: [2022-03-02 17:35:58,236] WARN [Consumer clientId=consumer-test_group-1, groupId=test_group] Connection to node 2147483646 (10.XX.XX.XX/10.XX.XX.XX:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2022-03-02 17:36:26,421] WARN [Consumer clientId=consumer-test_group-1, groupId=test_group] Connection to node 2 (10.XX.XX.XY/10.XX.XX.XY:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2022-03-02 17:36:47,467] WARN [Consumer clientId=consumer-test_group-1, groupId=test_group] Connection to node 3 (10.XX.XX.XZ/10.XX.XX.XZ:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient

Now some of the questionaries are ::

  1. Even if I am using Public IP to connect to Kafka, Why it is showing Private IP in logs ?
  2. Should bootstrap servers in producer and bootstrap servers in consumer must be exactly same ?
  3. How can I check the response of metadata request. Does kafka have any program to check ? Like it has for kafka-console-consumer.
  4. How should I fix this ?
  • Consider SASL and SSL properties and certificate in place.

It would be great if someone help me on this

Thanks, Milan K

Milan Kothari
  • 1
  • 2