zookeeper server due unable to load database on disk

Question

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry

We get the following error on one of the zookeeper server

[2019-11-12 07:44:20,719] ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer)
java.io.IOException: Unreasonable length = 198238896
               at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
               at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
               at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
               at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:629)
               at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)
               at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
               at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:601)
               at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:591)
               at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:164)
               at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
               at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

seems that some snapshot files under folder /opt/confluent/zookeeper/data/version-2 are corrupted

under folder version-2 , we have the following example files

many files as log.3000667b5
many files as snapshot.200014247
one file - acceptedEpoch
one file – currentEpoch

so the question is – how to start the zookeeper server

from my understanding we have two options , but not sure about them

1) one option is to move version-2 folder to other place as version-2_backup and create new folder - version-2 under /opt/confluent/zookeeper/data then start the zookeeper server and hope that snapshot will copied from other good active zookeeper server ?

2) second option is maybe to move version-2 folder to other place as version-2_backup , create new folder as - version-2 and copy all content from version-2 from good machine to the bad zookeeper server to version-2 , but I not sure if this is right option?

Maybe a bit late for you, but this happened to me just now and all the servers were screaming with the same error. How do you know which server has the non-corrupted data? Thanks a lot! — TanisDLJ, Oct 15 '20 at 16:20
Currently I am having the same problem. Were you sucessful with any of these two option. Would you bother to share how you solved the problem? — Jürgen Zornig, Mar 24 '21 at 10:17

zookeeper server due unable to load database on disk

0 Answers0