I have a Task Definition called kafka-zookeeper
, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and zookeeper containers from other kafka-zookeeper tasks, which is what I don't know how to do.
Basically, I need this:
------------------------
___________________ | ___________________ | ___________________
| Task1 | | | Task2 | | | Task2 |
|(kafka-zookeeper)| | |(kafka-zookeeper)| | |(kafka-zookeeper)|
| _____________ | | | _____________ | | | _____________ |
| | |<------ | | | | --->| | |
| | Kafka |<-------->| Kafka |<-------->| Kafka | |
| |___________| | | |___________| | | |___________| |
| ^ | | ^ | | ^ |
| | | | | | | | |
| _____v_______ | | ______v______ | | ______v______ |
| | | | | | | | | | | |
| | Zookeeper |<-------->| Zookeeper |<-------->| Zookeeper | |
| |___________|<------ | |___________| | --->|___________| |
|_________________| | |_________________| | |_________________|
| |
------------------------
But I don't know how to setup the communication between the tasks since the tasks' private IPs are determined once they're started. Also, if a task fails for some reason, when another one is created in its place the IP will be different. How do I deal with this?
I also think it's worth mentioning that zookeeper and kafka know about other instances by environment variables containing the IPs of the other instances.
I also can't give hostnames to containers because I'm using the awsvpc
network mode.
Thanks in advance!