14

I've got a one-node Cassandra cluster which is currently stuck in a major compaction process. After executing a nodetool compactit started the compaction and I see the tmp-sstable files with a size of 0 bytes. But that's all. No progress for hours.

I already stopped the Cassandra service, but after a restart it continued the compaction again without progress. A nodetool compactionstats shows 0,00% progress and a remaining time of six minutes. But I already waited 24 hours. In the meantime I stopped all readers and writers without seeing any difference.

The version I use is 2.1.2. I already tried with OpenJDK 7 and with Oracle's Java 7, but no difference.

The logs show no indication of OOM or any other exception or warning.

I'm not sure if it is of any help, but I currently migrate away from Cassandra as it doesn't fit my needs. So I delete a lot of data which has been migrated. There are many rows with several hundred up to a few thousand columns. But there are a handful rows with a million of columns. They now have a lot of tombstones in them due to the migration (deletion) process.

Any help in how to debug the issue are welcome.

mailq
  • 16,882
  • 2
  • 36
  • 66
  • just an idea, have you looked at /var/log/syslog or where would be logged something like to many files open/VM maxmem? – Antony Gibbs Dec 02 '14 at 00:11
  • @AntonyGibbs Nothing like that. It is an overpowered physical server. Other services run smoothly beside Cassandra. – mailq Dec 02 '14 at 19:10
  • 1
    Have you checked whether the process is CPU bound, I/O bound, or blocked waiting for something else? – kasperd Dec 03 '14 at 23:35
  • 1
    what are the settings of the launched java virtual machine? could be a powerfull server, if java instance is limited to 128M, it won't get more. The sound of nothing in logs makes me think memory limits... – Antony Gibbs Dec 06 '14 at 03:11
  • 1
    output of `ulimit -a` from a Cassandra-user console may be handy. – Antony Gibbs Dec 06 '14 at 03:19

1 Answers1

1

I'am not a cassandra expert, but did you tried nodetool stop compaction and to set a threshold for compaction (setcompactionthreshold to 0), so the system will not retry it

RichVel
  • 3,524
  • 1
  • 17
  • 23
yagmoth555
  • 16,300
  • 4
  • 26
  • 48
  • nodetool stop -- COMPACTION It didnt help. Jobs are still going on. – Jigar Shah Jul 31 '17 at 14:07
  • @JigarShah Same question as the OP? you should maybe ask a new question to attract new answer. As maybe my solution dont help you, but helped the OP – yagmoth555 Jul 31 '17 at 14:12