I have a 3 broker kafka cluster set up in an AWS VPC with kubernetes, I am using the https://github.com/Yolean/kubernetes-kafka repo for this.
I'm having the strange issue of being able to connect internally, produce and consume messages fine. When I connect externally, through a lot of debigging with kafkacat I can see that the initial connection to the brokers works, the metadata advertised listeners is correct, and those other brokers will even change there status to UP.
However as soon as I attempt to consume from a topic externally through kafkacat. Suddenly the correct addresses could not be resolved.
The brokers are listening externally on a CNAME aliased to an AWS LoadBalancer. This is on a High Availability VPC, so the Load Balancers have 3 IPs even though a pod will only be using one? (Not a network engineer or devops guy, just a software engineering student...).
Here is an exact example (created on my cluster, changed domain names):
user@comp:~/kubernetes-kafka$ kafkacat -L -b broker0.my-domain:9094 -t test-kafkacat
Metadata for test-kafkacat (from broker -1: broker0.my-domain:9094/bootstrap):
3 brokers:
broker 0 at broker0.my-domain:9094
broker 2 at broker2.my-domain:9094
broker 1 at broker1.my-domain:9094
1 topics:
topic "test-kafkacat" with 3 partitions:
partition 0, leader 0, replicas: 0,1, isrs: 0,1
partition 2, leader 2, replicas: 2,0, isrs: 2,0
partition 1, leader 1, replicas: 1,2, isrs: 1,2
user@comp:~/kubernetes-kafka$ kafkacat -C -b broker0.my-domain:9094 -t test-kafkacat
% ERROR: Local: Host resolution failure: broker2.my-domain:9094/2: Failed to resolve 'broker2.my-domain:9094': Name or service not known
% ERROR: Local: Host resolution failure: broker0.my-domain:9094/0: Failed to resolve 'broker0.my-domain:9094': Name or service not known
% ERROR: Local: Host resolution failure: broker1.my-domain:9094/1: Failed to resolve 'broker1.my-domain:9094': Name or service not known
And an example of me using internal addresses to consume messages from the same cluster:
user@comp:~/kubernetes-kafkakubectl --namespace test-kafka exec kafkacat-bk9rt -- kafkacat -C -b bootstrap.kafka:9092 -t test-kafkacat
Defaulting container name to producer.
Use 'kubectl describe pod/kafkacat-bk9rt -n test-kafka' to see all of the containers in this pod.
26:18,943654404+00:00 ---
Test kafkacat-bk9rt@2019-01-16T13:26:34,574792563+00:00
Test kafkacat-bk9rt@2019-01-16T13:26:44,537682179+00:00
Test kafkacat-bk9rt@2019-01-16T13:26:54,534127421+00:00
Test kafkacat-bk9rt@2019-01-16T13:27:04,532323895+00:00
Test kafkacat-bk9rt@2019-01-16T13:27:14,534851898+00:00
