0

I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and zookeeper containers from other kafka-zookeeper tasks, which is what I don't know how to do.

Basically, I need this:

                     ------------------------
___________________  | ___________________  | ___________________
|      Task1      |  | |      Task2      |  | |      Task2      |
|(kafka-zookeeper)|  | |(kafka-zookeeper)|  | |(kafka-zookeeper)|
| _____________   |  | | _____________   |  | | _____________   |
| |           |<------ | |           |   |  --->|           |   |
| | Kafka     |<-------->| Kafka     |<-------->| Kafka     |   |
| |___________|   |    | |___________|   |    | |___________|   |
|      ^          |    |       ^         |    |       ^         |
|      |          |    |       |         |    |       |         |
| _____v_______   |    | ______v______   |    | ______v______   |
| |           |   |    | |           |   |    | |           |   |
| | Zookeeper |<-------->| Zookeeper |<-------->| Zookeeper |   |
| |___________|<------ | |___________|   |  --->|___________|   |
|_________________|  | |_________________|  | |_________________|
                     |                      |
                     ------------------------

But I don't know how to setup the communication between the tasks since the tasks' private IPs are determined once they're started. Also, if a task fails for some reason, when another one is created in its place the IP will be different. How do I deal with this?

I also think it's worth mentioning that zookeeper and kafka know about other instances by environment variables containing the IPs of the other instances.

I also can't give hostnames to containers because I'm using the awsvpc network mode.

Thanks in advance!

ItsaMeTuni
  • 131
  • 1
  • 4

1 Answers1

0

I have found that it is not possible to do that with the awsvpc network mode. The way I've found to make this work was by following this article, which uses a Lambda function that updates Route53 records every time a new task is launched. I have made a java implementation of that here.

ItsaMeTuni
  • 131
  • 1
  • 4