16

Below the steps I did to get this issue :

  1. Launch ZooKeeper
  2. Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties

Also note that my OS is Windows

And at the second step the error happens :

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InconsistentClusterIdException: The Cluster ID Reu8ClK3TTywPiNLIQIm1w doesn't match stored clusterId Some(BaPSk1bCSsKFxQQ4717R6Q) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong. at kafka.server.KafkaServer.startup(KafkaServer.scala:220) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:84) at kafka.Kafka.main(Kafka.scala)

When I trigger .\bin\windows\kafka-server-start.bat .\config\server.properties zookeeper console returns :

INFO [SyncThread:0:FileTxnLog@216] - Creating new log file: log.1

How to fix this issue to get kafka running ?

EDIT See this answer following those steps solved my problem

TourEiffel
  • 261
  • 1
  • 2
  • 5
  • I suspect it's a problem with Kafka 2.4.0: https://stackoverflow.com/questions/59592518/kafka-broker-doesnt-find-cluster-id-and-creates-new-one-after-docker-restart – vanthome Jan 08 '20 at 13:32

9 Answers9

15

I solved this issue on my machine (Windows 10) by deleting the meta.properties file found in the log directory.

The issue itself occured after I had stopped both the zookeeper and server and then restarted them.

Jakob
  • 251
  • 2
  • 3
  • the logs are in /tmp directory – BJ5 May 23 '20 at 18:03
  • Good for you @johnmin The file I removed still was in the logs directory. – Jakob May 25 '20 at 06:39
  • Yes, it will get generated again, but Kafka will start without issues – BJ5 May 27 '20 at 14:29
  • @johnmin ok. I never claimed it wouldn't. Obviously the old file was the problem and the newly generated one worked fine. The removed file was in the logs directory. – Jakob Jun 24 '20 at 09:20
9

Usually, this problem arises when kafka logs are stored in the persistent folder and zookeeper data in the temporary, or vice-versa. Then, after system restart, files that are stored in the temporary directory get cleaned and regenerated leading to the configuration mismatch.

To prevent this situation to happen again revise log.dirs kafka parameter in the server.properties and dataDir zookeeper parameter in zookeeper.properties and ensure that both point to the same type of directory (e.g. temporary or persistent). Only after that reset inconsistent data (if any) using for example method from Jacob's answer, i.e. remove meta.properties file, and restart zookeeper and kafka.

fivef
  • 1,369
  • 1
  • 8
  • 5
hovercraft
  • 91
  • 1
  • 2
3

I encountered the same issue while running Kafka server on my Windows Machine. You can try the following to resolve this issue:

  1. Open server server.properties file which is located in your kafka folder kafka_2.11-2.4.0\config (considering your version of kafka, folder name could be kafka_<kafka_version>)
  2. Search for entry log.dirs
  3. If your log.dir path contains windows directory path like this E:\Shyam\Software\kafka_2.11-2.4.0\kafka-logs which has a single backslash i.e , change it to double back-slash i.e with \
Kalle Richter
  • 259
  • 6
  • 17
Shyam Yadav
  • 31
  • 2
1

Try the following...

  1. Enable following line in ./config/server.properties

    listeners=PLAINTEXT://:9092

  2. Modify default ZooKeeper dataDir

  3. Modify default Kafka log dir

user558910
  • 11
  • 1
1

On windows version 20H2 and OS build 19042.662 escaping the log.dirs path in <kafka_dir>\config directory worked for me.

it was previously

log.dirs=C:\*********\kafka\logs

changed to

log.dirs=C:\\*********\\kafka\\logs

Might need to restart zookeeper first and then kafka after the change.

deepugn
  • 11
  • 4
1

No need to delete the log/data files on Kafka. Check the Kafka error logs and find the new cluster id. Update the meta.properties file with cluster-ID then restart the Kafka.

/home/kafka/logs/meta.properties

To resolve this issue permanently follow below.

Check your zookeeper.properties file and look for dataDirpath and change the path tmp location to any other location which should not be removed after server restart.

/home/kafka/kafka/config/zookeeper.properties

Copy the zookeeper folder and file to the new(below or non tmp) location then restart the zookeeper and Kafka.

cp -r /tmp/zookeeper /home/kafka/zookeeper

Now server restart won’t affect the Kafka startup.

Aditya Y
  • 121
  • 3
0

Try to delete kafka-logs from your tmp folder, which is located in C:/tmp folder by default on windows, and can be set in [your-kafka-folder]/config/server.properties file as log.dirs

Peter Matyas
  • 1
0

Delete /tmp/kafka-logs/meta.properties which saved the wrong cluster.id of last/failed session:

#
broker.id=0
version=0
cluster.id=kzRHvVl-Rb2DZaHSqlTfow     <===
Haili Sun
  • 101
  • 1
0

If you do not mind losing your data than you can remove your kafka "data" folder and restart kafka. If you used the default location the folder named "data" is probably in /tmp/kafka-logs.

thanksalot
  • 1
  • Thank you for your reply, the path of my log file in server.properties is `log.dirs=C:\\Users\\Dorian\\kafka_2.12-2.4.0\\kafka-logs` is it wrong ? – TourEiffel Jan 06 '20 at 10:30
  • I also have no data folder into `kafka_2.12-2.4.0` so how should i set up the data folder path ? – TourEiffel Jan 06 '20 at 10:46