5

I'm having a hard time restoring snapshot on Apache Cassandra (version 3.0.9). As far as I can say, I'm following the procedure described on datastax blog, along with several other ones (for instance : http://datascale.io/cloning-cassandra-clusters-fast-way/). Yet I may be missing something, and everytime I make a restore, data is missing.

Setup : 6 nodes cluster (1 DC, 3 racks with 2 nodes each) with a replication factor set to 3. Machines are hosted on AWS.

Backup procedure (on each node) :

  1. nodetool snapshot mykeyspace
  2. cqlsh -e 'DESCRIBE KEYSPACE mykeyspace' > /tmp/mykeyspace.cql
  3. nodetool ring | grep "$(ifconfig | awk '/inet /{print $2}' | head -1)" | awk '{print $NF ","}' | xargs > /tmp/tokens

I get the files generated by the nodetool snapshot command and backup them along with tokens and cql on S3.

Restore procedure (for each node unless it's specified) :

(after having created new VMs)

  1. Download snapshots, tokens and keyspace
  2. Stop service cassandra
  3. Delete /var/lib/cassandra/commitlog/* and /var/lib/cassandra/system/
  4. Insert tokens into cassandra.yaml
  5. Start service cassandra
  6. Restore mykeyspace from mykeyspace.cql on one node only
  7. Wait for replication and stop service cassandra
  8. Delete .db files in folder /var/lib/cassandra/data/mykeyspace/
  9. For each table copy snapshots files (.db, .crc32, .txt) into /var/lib/cassandra/data/mykeyspace/$table/
  10. Restart service cassandra
  11. Run nodetool repair mykeyspace -full, one node at a time

Result :

There are always missing rows, approximately the same quantity for each table but never the same ones. I tried to "mix up" a bit the procedure, like restoring keyspace before tokens, running nodetool refresh before repair, but I meet the same issue everytime.

Since I'm not far from having a "good" restore, I think that I'm missing something pretty obvious. Analyzing logs didn't really help, as they don't show any error/failure messages.

Any help would be welcomed :) I can of course give more information if needed.

edit : no one ? I updated the question with cassandra version (3.0.9), which I forgot in the first place. I tried again to restore, but no luck. I don't have any more idea really :(

P. Bender
  • 51
  • 1
  • 5

2 Answers2

0

The sed command in that blog post, which is supposed to append -Dcassandra.load_ring_state=false to the $JVM_OPTS, has no effect in its current form.

If you were copying that command directly from the blog post, it's possible that's the issue. You could try this one instead which places it at the bottom of the file:

sudo sed -i '$ a\JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"' /etc/cassandra/cassandra-env.sh

Also you'll need to do a nodetool repair -pr <ks> on each node, one by one, after following this procedure.

Josh
  • 101
  • 3
  • Thanks for your answer :) But unfortunately, it didn't improve my restore. I'm going to try the tablesnap / tableslurp utility (https://github.com/JeremyGrosser/tablesnap) and see if it's more accurate than my solution. – P. Bender Nov 15 '16 at 15:12
  • 1
    You also might double check that the `intial_token` setting on each node aligns with the tokens on the node from which the sstables were copied. On my first attempt before fully automating this, I think I swapped one and had missing data as you described above. It was a single key, in one table, but a different key each time I tried to restore with the tokens out of order. Good luck either way! – Josh Nov 15 '16 at 15:34
0

Ok, end of story, stupid me ! The initial_token line in cassandra.yaml was wrongly "seded" during my restore procedure. If there is no space after the ':' for the initial_token key, then cassandra fails to launch. therefore the line was kept commented and tokens not interpreted !

tldr :

  • initial_token:<values> = WRONG
  • initial_token: <values> = GOOD

Thanks a lot to you Josh Purvis for insisting on the high importance of this parameter :-)

P. Bender
  • 51
  • 1
  • 5