I have a Redis cluster with 3 masters and 3 slaves. I can add data to the cluster and get data back, no problems.
The issue comes when the data is saved to a master node. After a while the master will restart itself (see output below) and then the data it held will be lost. If the cluster decides to use a slave to store the data on then the data is persisted until the ttl is reached.
When the restart occurs the output in the logs is as follows:
30580:signal-handler (1479922724) Received SIGTERM scheduling shutdown...
30580:M 23 Nov 17:38:44.544 # User requested shutdown...
30580:M 23 Nov 17:38:44.545 * Calling fsync() on the AOF file.
30580:M 23 Nov 17:38:44.545 * Saving the final RDB snapshot before exiting.
30580:M 23 Nov 17:38:44.567 * DB saved on disk
30580:M 23 Nov 17:38:44.567 * Removing the pid file.
30580:M 23 Nov 17:38:44.567 # Redis is now ready to exit, bye bye...
30636:M 23 Nov 17:38:44.783 * Increased maximum number of open files to 10032 (it was originally set to 1024).
17513:M 23 Nov 17:31:10.552 * Node configuration loaded, I'm cd698d5a4350c161930b40c52f71e1fabeac549d
<%insert_logo_and_pid_here%>
17513:M 23 Nov 17:31:10.553 # Server started, Redis version 3.2.4
17513:M 23 Nov 17:31:10.553 * The server is now ready to accept connections on port 6379
17513:M 23 Nov 17:31:11.404 * Slave <%ip%>:6379 asks for synchronization
17513:M 23 Nov 17:31:11.404 * Full resync requested by slave <%ip%>:6379
17513:M 23 Nov 17:31:11.404 * Starting BGSAVE for SYNC with target: disk
17513:M 23 Nov 17:31:11.405 * Background saving started by pid 17518
17518:C 23 Nov 17:31:11.418 * DB saved on disk
17518:C 23 Nov 17:31:11.418 * RDB: 0 MB of memory used by copy-on-write
17513:M 23 Nov 17:31:11.455 * Background saving terminated with success
17513:M 23 Nov 17:31:11.455 * Synchronization with slave <%ip%>:6379 succeeded
17513:M 23 Nov 17:31:12.557 # Cluster state changed: ok
I am not sure if this is something killing it or whether it is saving itself and restarting? I can see there is the sigterm and user requested shutdown but I have not done anything of the such.
Also the eagle eyed amongst you will notice the difference in line numbers, this is because I copied it from a different Redis instance and couldn't be bothered to change it.
The server it is running on is Centos 7, Redis version 3.2.4.
Any help would be greatly appreciated, thanks in advance.
