Issue with snapshot creation

Question

I am running into an issue when I try to repair a keyspace:

myuser@host-1:~$ nodetool repair -pr mykeyspace
[2014-11-07 13:33:10,132] Starting repair command #13, repairing 1 ranges for keyspace mykeyspace
[2014-11-07 13:33:20,668] Repair session 3baeeb40-667a-11e4-a6e9-3562b4c30e9e for range (-3651079129967449,-2] failed with error java.io.IOException: Failed during snapshot creation.
[2014-11-07 13:33:20,668] Repair command #13 finished

When I look in system.log on host-1 it says:

ERROR [RepairJobTask:1] 2014-11-07 13:33:20,665 RepairJob.java (line 125) Error occurred during snapshot phase
java.lang.RuntimeException: Could not create snapshot at /X.X.X.33
        at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:81)
        at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:344)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [AntiEntropySessions:12] 2014-11-07 13:33:20,666 RepairSession.java (line 288) [repair #3baeeb40-667a-11e4-a6e9-3562b4c30e9e] session completed with the following error
java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
ERROR [AntiEntropySessions:12] 2014-11-07 13:33:20,667 CassandraDaemon.java (line 199) Exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        ... 3 more
ERROR [Thread-1607] 2014-11-07 13:33:20,668 StorageService.java (line 2599) Repair session 3baeeb40-667a-11e4-a6e9-3562b4c30e9e for range (-3651079129967449,-2] failed with error java.io.IOException: Failed during snapshot creation.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2591)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Failed during snapshot creation.
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        ... 1 more
Caused by: java.io.IOException: Failed during snapshot creation.
        at org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:323)
        at org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:126)
        at com.google.common.util.concurrent.Futures$4.run(Futures.java:1160)
        ... 3 more

When I look in the log on X.X.X.33 I don't see any log errors whatsoever.

Anyone who could give me pointer in the right direction?

A good guess, yes. I should have made it clear that I had checked the disk space. No disk full. I found the solution. See my answer below. — Ztyx, Nov 07 '14 at 13:43

score 2 · Accepted Answer · answered Nov 07 '14 at 13:32

Fixed. I issued a manual nodetool snapshot mykeyspace on X.X.X.33 and managed to get a better error in the log. Turns out I had a missing datafile, probably from dropping a table and creating it faster than that schema change had propagated throughout the cluster. Restarted the Cassandra node (for it to discover the file was missing) and TRUNCATEd the table.

Issue with snapshot creation

1 Answers1