Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

94 questions
16
votes
9 answers

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Below the steps I did to get this issue : Launch ZooKeeper Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties Also note that my OS is Windows And at the second step the error happens : ERROR Fatal error during…
TourEiffel
  • 261
  • 1
  • 2
  • 5
8
votes
3 answers

Kafka logfile location

Just to make some things clear: I'm talking about the process logfile that contains the stdout and stderr messages. This is my systemd unit file: [Unit] Description=Apache Kafka server Documentation=http://kafka.apache.org Requires=network.target…
Jeroen Jacobs
  • 1,276
  • 3
  • 15
  • 24
4
votes
1 answer

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: - 1) Generate certificate for each broker kafka: COMANDO: keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey - 2) Create CA. The…
xRobot
  • 141
  • 1
  • 1
  • 4
4
votes
1 answer

Kafka compatible with Zookeeper 3.5 feature 'Rebalancing Client Connections'

In this document https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html dynamic configuration functionality is described for Zookeeper 3.5. There are 2 important points in this document: When changing the dynamic config of one Zookeeper…
hendrikswan
  • 141
  • 5
4
votes
1 answer

Choose number of partitions in Kafka

I read the documentation of apache kafka but I couldn't find an example about how many partitions should I use in any scenario. For example lets say that I have 5000 msgs/entries per minute, for this situation how many partitions should I have (or…
Diego Velez
  • 780
  • 1
  • 6
  • 13
4
votes
0 answers

Kafka using a distant Zookeeper instance?

we have an existing Zookeeper instance, and want to build a Kafka cluster in a remote location (not on the same side of the Atlantic ocean). Can we use that existing Zookeeper for our new Kafka usage ? Would latency be an issue ? Kafka is using…
Pixou
  • 141
  • 2
3
votes
2 answers

Enable logical replication on Google Cloud Postgres

Is there anyway to enable logical replication wal_level = logical on Google Cloud Postgres instance? I want to do CDC (change data capture) to stream WAL Record to Apache Kafka. Thanks for your help!
Kevin
  • 133
  • 4
3
votes
1 answer

Does Kafka replicate the same data across all brokers?

I am going to implement Kafka in the infrastructure. I have a use case where we have below components in the Kafka architecture. Number of brokers: 5 Replication Factors: 3 Number of topics: 3 Number of partition: 3 I have to implement scaling…
mohit
  • 181
  • 8
3
votes
2 answers

Apache Kafka configuration behind NAT/LB

In order to make our Kafka cluster available both from Internet and from our private network, we thought fine to configure Kafka this way : Private VIP:9000 => All brokers:9092 (topology query only) Private VIP:9001 => Broker #1:9092 Private…
mveroone
  • 447
  • 7
  • 22
3
votes
1 answer

Running Kafka in Kubernetes on AWS

I've the following scenario: 3 nodes instances spread across 3 Availability Zones 6 Brokers running in StatefulSets with AWS external volume for Kafka data. Instance size: m4.2xlarge EBS Volume: st1 - 500 GiB No limit requests and memory set at…
Mazzy
  • 205
  • 2
  • 11
3
votes
1 answer

How to read back collectd and write_kafka plugin

I want to use kafka as a transport layer for collectd. I found that there is a write_kafka plugin for collectd, which sends all the gathered metrics to a kafka topic. My intention was to have a few hosts as collectors (working as a kafka consumer…
Jeroen Jacobs
  • 1,276
  • 3
  • 15
  • 24
3
votes
2 answers

Kafka stores log4j logs in directory literally called ${kafka.logs.dir}

I installed a preview version of Kafka 3.0 with log4j2 support (http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/) on a RHEL 8 server. Kafka and Zookeeper are running successfully as systemd user services. I set the environment…
Cabbage Parachute
  • 33
  • 5
2
votes
1 answer

Linux + docker container folder that mounted to OS volume + how to know the real limited size

inside the docker-compose.yml we configured the following volume image: confluentinc/cp-kafka:latest volumes: - /grid/kafka-data:/var/lib/kafka/data . docker-compose ps Name Command State …
shalom
  • 451
  • 12
  • 26
2
votes
1 answer

Kafka log retention not wokring as expected

Note: I am aware that there are many similar questions, already looked at them, didn't help. I am running a kafka benchmark test in a closed environment, I gave each broker a filesystem of 40GB for log persistence, and I'm quickly getting to a…
Tom Klino
  • 601
  • 1
  • 7
  • 14
2
votes
3 answers

ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

I have zookeeper.version=3.4.10, and I see: https://issues.apache.org/jira/browse/ZOOKEEPER-2307 that is however in unresolved status. The problem appeared because of filling up the disk space. Now that I deleted some logs, and freed space, I would…
hdjur_jcv
  • 191
  • 1
  • 4
1
2 3 4 5 6 7