1

I've set up four server and sentinel instances on four different virtual machines in a simple master-slave setup. The key replication works fine, and all redis nodes recognized each other (I'm referring to the known-sentinel and known-slave options).

The problem comes up when I'm trying to simulate a failure. The sentinels just keep telling that they can't elect a new master (failover-abort-not-elected):

12481:X 22 Jun 17:20:05.761 # <__IP1__>:51002 voted for a3ba8d8cd0c15c8b186b44ba529b83d3a8424a42 27
12481:X 22 Jun 17:20:14.140 # -sdown master mymaster <__IP2__> 6379
12481:X 22 Jun 17:20:14.140 # -odown master mymaster <__IP2__> 6379
12481:X 22 Jun 17:20:16.567 # -failover-abort-not-elected master mymaster <__IP2__> 6379

Another strange thing that when I query the sentinel's state in redis-cli, I keep getting greater values for the number of available sentinel instances than the actual count (which is of course, cannot be greater then 4 (right?)).

$ redis-cli -p 51001
> info
[...]
master0:name=mymaster,status=ok,address=91.120.59.231:6379,slaves=3,sentinels=8

Also, I keep getting these -dup-sentinel messages when starting up sentinel:

14563:X 23 Jun 11:10:18.015 * -dup-sentinel master mymaster <__IP_MASTER__> 6379 #duplicate of <__IP_SLAVE2__>:51003 or 65db9ddba84433f71aa77c9263807e6abd939d2c

Here's one of the sentinels' config:

daemonize yes
pidfile "/var/run/redis/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"

# bind 192.168.1.100 10.0.0.1
# port 26379

port 51004

dir "/var/lib/redis"

sentinel monitor mymaster <__IP_MASTER__> 6379 2
sentinel failover-timeout mymaster 60000
sentinel auth-pass mymaster <___________PASSW___________>
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 55428

# Generated by CONFIG REWRITE
maxclients 4064
sentinel known-slave mymaster <__IP_SLAVE1__> 6379
sentinel known-slave mymaster <__IP_SLAVE2__> 6379
sentinel known-slave mymaster <__IP_SLAVE3__> 6379
sentinel known-sentinel mymaster <__IP_SLAVE1__> 51002 bfea8e89873353d5c1e9ed498b17f298bd6b6082
sentinel known-sentinel mymaster <__IP_MASTER__> 26379 c04fc07022a251e56e81e21dbe018bd471745038
sentinel known-sentinel mymaster <__IP_SLAVE2__> 51003 b5a56eb69abb5893203ecf7b1a7d28ae4aa06ab5
sentinel known-sentinel mymaster <__IP_MASTER__> 51001 be1ff6c9e4a62abea132c24447fd3cb38d221496
sentinel known-sentinel mymaster <__IP_SLAVE1__> 26379 f5a7fd0ee199afc02ffafc0e822363acf5595b1c
sentinel known-sentinel mymaster <__IP_SLAVE2__> 26379 6328962293b7838fa88b9f176b9bc4cc08f2e26d
sentinel current-epoch 26

The log files aren't really informative... is there a way to make it a little bit more verbose? Do you have any idea what am I misssing here?

Thanks in advance!

Bumping post:

Checked firewall settings, it should have been okay, I was still getting the same error when I shut off the firewall completley.

Gergely Lukacsy
  • 111
  • 1
  • 6

2 Answers2

2

Taken from sentinel default config:

# *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1

So try to bind your server's IP.

Gluck!

Justin
  • 121
  • 4
  • Something worth pointing out is that this can be super difficult to notice because the configs can get rewritten by sentinel and point out all the sentinels that are running and still not be able to perform an election. It's unclear where the "not be reachable" starts and ends. Thanks for posting this answer! Saved me a ton of effort :) – artburkart Jun 07 '17 at 20:32
  • @artburkart Glad I can help :) – Justin Jun 13 '17 at 03:59
1

Old sentinels are not removed from configuration. Look at docs redis sentinel

Execute: SENTINEL RESET mymaster and for a while, new master should be elected.

bu21
  • 11
  • 2