I have a node Redis cluser with 1 master and 2 slaves. Sentinel is running on all of them.
master 172.29.245.6
slave1 172.29.240.163
slave2 172.29.225.104
With my master up, this is the status of my cluster.
Version --> redis-3.2.10-2.el7.x86_64
Master
# Replication
role:master
connected_slaves:2
slave0:ip=172.29.225.104,port=6379,state=online,offset=486,lag=1
slave1:ip=172.29.240.163,port=6379,state=online,offset=633,lag=0
master_repl_offset:925
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:924
Slave1
# Replication
role:slave
master_host:172.29.245.6
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:17719
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
Slave2
# Replication
role:slave
master_host:172.29.245.6
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:20367
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
My aim is that when I shut down the master redis process one of the slaves should become a master.
But when I try to do an automatic failover, it gives the following messages.
Oct 27 14:18:22 redis-2 redis-sentinel[4866]: 4866:X 27 Oct 14:18:22.175 # +new-epoch 27
Oct 27 14:18:22 redis-2 redis-sentinel[4866]: 4866:X 27 Oct 14:18:22.175 # +try-failover master redis-master 172.29.245.6 6379
Oct 27 14:18:22 redis-2 redis-sentinel[4866]: 4866:X 27 Oct 14:18:22.185 # +vote-for-leader 61c51a37f2db1673ddcf5dc3fe6816c9ba83a408 27
Oct 27 14:18:33 redis-2 redis-sentinel[4866]: 4866:X 27 Oct 14:18:33.142 # -failover-abort-not-elected master redis-master 172.29.245.6 6379
Oct 27 14:18:33 redis-2 redis-sentinel[4866]: 4866:X 27 Oct 14:18:33.218 # Next failover delay: I will not start a failover before Fri Oct 27 14:20:23 2017
However when I use the method to manually failover, then it works.
[root@redis-1 ~]# redis-cli -p 26379 SENTINEL failover redis-master
OK
Oct 27 14:19:40 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:19:40.503 # -sdown master redis-master 172.29.245.6 6379
Oct 27 14:19:40 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:19:40.503 # -odown master redis-master 172.29.245.6 6379
Oct 27 14:24:56 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:24:56.982 # +new-epoch 28
Oct 27 14:24:57 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:24:57.851 # +config-update-from sentinel 5080eb7a0e27f9fb6ea6a197bf1abd334608943f 172.29.245.6 26379 @ redis-master 172.29.245.6 6379
Oct 27 14:24:57 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:24:57.852 # +switch-master redis-master 172.29.245.6 6379 172.29.225.104 6379
Oct 27 14:24:57 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:24:57.852 * +slave slave 172.29.240.163:6379 172.29.240.163 6379 @ redis-master 172.29.225.104 6379
Oct 27 14:24:57 redis-3 redis-sentinel[1211]: 1211:X 27 Oct 14:24:57.852 * +slave slave 172.29.245.6:6379 172.29.245.6 6379 @ redis-master 172.29.225.104 6379
I have gone through the documentation and followed it exactly but the automatic failover for me does not work. Any ideas ?
My sentinel.conf on all the nodes look like:
MAster
port 26379
sentinel myid 5080eb7a0e27f9fb6ea6a197bf1abd334608943f
sentinel monitor redis-master 172.29.245.6 6379 1
sentinel down-after-milliseconds redis-master 5000
sentinel failover-timeout redis-master 60000
sentinel auth-pass redis-master XXXXXXXXXXXXXXXXX
# Generated by CONFIG REWRITE
dir "/"
sentinel config-epoch redis-master 29
sentinel leader-epoch redis-master 29
sentinel known-slave redis-master 172.29.225.104 6379
sentinel known-slave redis-master 172.29.240.163 6379
sentinel known-sentinel redis-master 172.29.240.163 26379 61c51a37f2db1673ddcf5dc3fe6816c9ba83a408
sentinel known-sentinel redis-master 172.29.225.104 26379 85e2ce3707304f7b3b4d0f7281dcd6932094c8b6
sentinel current-epoch 29
Slave1
port 26379
sentinel myid 61c51a37f2db1673ddcf5dc3fe6816c9ba83a408
sentinel monitor redis-master 172.29.245.6 6379 1
sentinel down-after-milliseconds redis-master 5000
sentinel failover-timeout redis-master 60000
sentinel auth-pass redis-master XXXXXXXXXXXXXXXX
# Generated by CONFIG REWRITE
dir "/"
sentinel config-epoch redis-master 29
sentinel leader-epoch redis-master 27
sentinel known-slave redis-master 172.29.225.104 6379
sentinel known-slave redis-master 172.29.240.163 6379
sentinel known-sentinel redis-master 172.29.245.6 26379 5080eb7a0e27f9fb6ea6a197bf1abd334608943f
sentinel known-sentinel redis-master 172.29.225.104 26379 85e2ce3707304f7b3b4d0f7281dcd6932094c8b6
sentinel current-epoch 29
Slave2
port 26379
sentinel myid 85e2ce3707304f7b3b4d0f7281dcd6932094c8b6
sentinel monitor redis-master 172.29.245.6 6379 1
sentinel down-after-milliseconds redis-master 5000
sentinel failover-timeout redis-master 60000
sentinel auth-pass redis-master XXXXXXXXXXXX
# Generated by CONFIG REWRITE
dir "/"
sentinel config-epoch redis-master 29
sentinel leader-epoch redis-master 26
sentinel known-slave redis-master 172.29.225.104 6379
sentinel known-slave redis-master 172.29.240.163 6379
sentinel known-sentinel redis-master 172.29.245.6 26379 5080eb7a0e27f9fb6ea6a197bf1abd334608943f
sentinel known-sentinel redis-master 172.29.240.163 26379 61c51a37f2db1673ddcf5dc3fe6816c9ba83a408
sentinel current-epoch 29
My Sentinel config also looks good:
[root@redis-1 ~]# redis-cli -p 26379 info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=redismaster,status=ok,address=172.29.245.6:6379,slaves=2,sentinels=3
