Would Kafka Producer & Consumer hosts can be same. Suppose if I've producers like host1,host2 & host3. Can I use same host1, host2 & host3 as consumers? What are the consequences/challenges if I use producers & consumers as same hosts.
Asked
Active
Viewed 103 times
1 Answers
0
Sure you can combine those. e.g. Kafka Streams processor do exactly that under the hood. Both are clients of the Kafka cluster, and depending on your used case and throughput requirements you can share servers for them. However you should have a clear view on network and disk I/O and CPU usage. I personally find it easier to manage dedicated machines, which allows me to examine causes without interferences in case of issues.
mjahr
- 3
- 4
-
I would like to understand how it behaves if one host goes down. Do we get any issues in leader elections or something? – KKE Jul 17 '19 at 20:06
-
Producers and Consumers are clients of Kafka brokers. They behave differently when they stop working, depending on the producer and consumer configuration. Consumers typically work in consumer groups, this means that Kafka is able to realing the load to remaining consumers. Producers generate messages, and when one fails, no more messages are generated. You have to ensure redundancy on the producer side on your own. When a broker fails, consumers and producers can try to connect to another broker if it is set up in their configuration. Leader management is a part of the brokers' concern only. – mjahr Jul 28 '19 at 00:08