3

I've the following scenario:

3 nodes instances spread across 3 Availability Zones

  • 6 Brokers running in StatefulSets with AWS external volume for Kafka data.
  • Instance size: m4.2xlarge
  • EBS Volume: st1 - 500 GiB
  • No limit requests and memory set at Kubernetes resource level (not good for production - resource limits must be set) 1 topic, 6 partitions, no replication. Used librdkafka with Kafka version 0.11.0

Producer sends a message of size 100 bytes

Here the command:

kubectl exec -it kafka-kafka-librdkafka -- examples/rdkafka_performance -P -t test -s 100 -b kafka-kafka-headless:9092 -X request.timeout.ms=900000 -X batch.num.messages=10000 -X queue.buffering.max.ms=1000

The result:

% Sending messages of size 100 bytes
% 500000 messages produced (50000000 bytes), 0 delivered (offset 0, 0 failed) in 1000ms: 0 msgs/s and 0.00 MB/s, 41 produce failures, 500000 in queue, no compression
% 1000000 messages produced (100000000 bytes), 500000 delivered (offset 0, 0 failed) in 2000ms: 249957 msgs/s and 25.00 MB/s, 65 produce failures, 500000 in queue, no compression
% 1525491 messages produced (152549100 bytes), 1025491 delivered (offset 0, 0 failed) in 3000ms: 341774 msgs/s and 34.18 MB/s, 90 produce failures, 500000 in queue, no compression
% 1958991 messages produced (195899100 bytes), 1525500 delivered (offset 0, 0 failed) in 4000ms: 381328 msgs/s and 38.13 MB/s, 120 produce failures, 433491 in queue, no compression
% 2232174 messages produced (223217400 bytes), 2028173 delivered (offset 0, 0 failed) in 5000ms: 405594 msgs/s and 40.56 MB/s, 150 produce failures, 204001 in queue, no compression
% 2622943 messages produced (262294300 bytes), 2528180 delivered (offset 0, 0 failed) in 6000ms: 421328 msgs/s and 42.13 MB/s, 161 produce failures, 94763 in queue, no compression
% 3145529 messages produced (314552900 bytes), 3035578 delivered (offset 0, 0 failed) in 7000ms: 433623 msgs/s and 43.36 MB/s, 176 produce failures, 109951 in queue, no compression
% 3675274 messages produced (367527400 bytes), 3498817 delivered (offset 0, 0 failed) in 8039ms: 435186 msgs/s and 43.52 MB/s, 196 produce failures, 176458 in queue, no compression
% 4181717 messages produced (418171700 bytes), 3961228 delivered (offset 0, 0 failed) in 9042ms: 438068 msgs/s and 43.81 MB/s, 213 produce failures, 220489 in queue, no compression
% 4669614 messages produced (466961400 bytes), 4499671 delivered (offset 0, 0 failed) in 10085ms: 446156 msgs/s and 44.62 MB/s, 230 produce failures, 169946 in queue, no compression
% 5071907 messages produced (507190700 bytes), 4964422 delivered (offset 0, 0 failed) in 11132ms: 445930 msgs/s and 44.59 MB/s, 230 produce failures, 107490 in queue, no compression
% 5638247 messages produced (563824700 bytes), 5392203 delivered (offset 0, 0 failed) in 12141ms: 444125 msgs/s and 44.41 MB/s, 231 produce failures, 246046 in queue, no compression

With the resource we have I think we are capped somewhere. Do you have any idea where it could be the bottleneck?

Mazzy
  • 205
  • 2
  • 11

1 Answers1

0

I manage 3 kubernetes clusters and 4 kafka clusters (3 kafka nodes each) at my job.

I would NEVER put kafka inside of kubernetes. It makes no sense at all.

Just use dedicated ec2 machines for kafka clusters, and connect k8s VPCs with kafka VPCs.

Kafka is a database, it benefits hugely from direct access to operating system, and you can tune it much more freely than k8s deployment.

K8s is not an universal hammer, use it for services and cronjobs and keep databases outside.

cohadar
  • 764
  • 7
  • 6
  • Although I appreciate your answer in this topic, it doesn't give any value to the discussion. It would be interesting to have data to support your thesis. Several solutions appeared in the last years to run stateful app like databases in kubernetes so at the moment I don't see any valuable argument which stop me to run Kafka in kubernetes – Mazzy Oct 30 '17 at 19:58