2

I have three servers with mongod installed on it running as a replication set. Suddenly the two secondories became unavailable (the mongod process died) - I think because they were too stale.

The problem is that the original PRIMARY is now the SECONDARY and my application doesn't work because it can't connect to a PRIMARY.

I mean, in which way does that help me? If the replica set can't do failover?! Am I missing something?

Furhtermore I am asking myself why did the SECONDARIES die / why are they too stale? What can I do about it?

FYI: My database is quite big (40GB on disk).

Mads
  • 133
  • 1
  • 1
  • 5

1 Answers1

4

The reason why your primary is now no longer primary is because you have to have enough votes to form a majority in a replica set - in your case that means 2/3 minimum. With your specific failure that doesn't seem to make sense, but imagine instead that the primary in question had instead become isolated from the 2 secondaries (a network event for example). In that case you would not want two primaries elected, so instead it would step down until it could see the other set members.

In order to get the primary back to working you need to do one of two things:

  1. Get the other set members back up
  2. Reconfigure the set so that a majority can be formed (add more members, go to a single)

See here for more information on the reconfiguration option:

http://www.mongodb.org/display/DOCS/Reconfiguring+a+replica+set+when+members+are+down

Basically, your replica set will failover, but you have to have enough members to actually do an election.

In terms of why the secondaries died, I can't say based on the information you have provided here, but I can tell you that it's not because they were too stale. A secondary that falls too far behind will report as being stale but will keep running (and voting for the primary) even in that state, until you fix it and bring it back up to date. See here for how to remedy a stale set member:

http://www.mongodb.org/display/DOCS/Resyncing+a+Very+Stale+Replica+Set+Member

Adam C
  • 5,132
  • 2
  • 28
  • 49