throwable caught when submitting- druid coordinator

0

I have upgraded my druid cluster from 0.12 to 0.16 recently. coordinator-overload log is flooded with bellow exception now.

2019-10-25T16:57:12,257 ERROR [Master-PeonExec--0] org.apache.druid.server.coordinator.CuratorLoadQueuePeon - Server[/data/druid/loadQueue/hdsn11.vxanalytics.com:8083], throwable caught when submitting [SegmentChangeRequestLoad{segment=DataSegment{binaryVersion=9, id=ipdr_hourly_usage_summarize_data_2_2019-09-29T07:00:00.000Z_2019-09-29T08:00:00.000Z_2019-09-29T02:57:57.362Z, loadSpec={type=>hdfs, path=>hdfs://VX-BI-Cluster/data/datadir/segments/ipdr_hourly_usage_summarize_data_2/20190929T070000.000Z_20190929T080000.000Z/2019-09-29T02_57_57.362Z/0_index.zip}, dimensions=[protocol_name, category, upload, download, total, cell_id, cell_name, cell_site, district, longitude, latitude, uri], metrics=[count, value], shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, size=45263397}}].
org.apache.druid.java.util.common.ISE: /data/druid/loadQueue/hdsn11.vxanalytics.com:8083/ipdr_hourly_usage_summarize_data_2_2019-09-29T07:00:00.000Z_2019-09-29T08:00:00.000Z_2019-09-29T02:57:57.362Z was never removed! Failing this operation!
    at org.apache.druid.server.coordinator.CuratorLoadQueuePeon$SegmentChangeProcessor.lambda$scheduleNodeDeletedCheck$1(CuratorLoadQueuePeon.java:285) ~[druid-server-0.16.0-incubating.jar:0.16.0-incubating]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_192]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_192]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_192]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]

Any idea what happens here ?

Shamin Weerarathna

Posted 2019-10-25T11:42:48.607

Reputation: 9

No answers