Results at a glance
- 5voting members: primary, secondaries and an arbiter
- Autoprimary election when a member fails
- Prioritiescontrol which node becomes primary
- Runbooksfor maintenance and disaster recovery
The goal
Applications on the platform write to MongoDB all day. A single server would be a single point of failure, so the database needed to keep working when any one machine went down.
What I built
- Replica set with an arbiter. Data-bearing members (one primary, the rest secondaries) plus an arbiter as an extra vote, giving five voting members so elections always have a clear majority.
- Member priorities. Replica set priorities decide which member is preferred as primary, so after a failover and recovery the set returns to a known layout.
- Read preferences. Reads that can tolerate slightly stale data go to secondaries, which takes load off the primary.
- Monitoring. Replication lag and member health in Grafana, with alerts before a secondary falls too far behind to be a safe failover target.
- Runbooks. Step-by-step procedures for rolling maintenance (secondaries first, then step the primary down) and for recovering a failed member.
Decisions and trade-offs
- Arbiters are cheap but not free. An arbiter votes but holds no data. If too many data-bearing members are down, majority write concern can stall even though an election succeeds. That's why the member layout and write concerns have to be planned together, not added later.
- Secondary reads are eventually consistent. Only reads that can tolerate that go to secondaries. Anything that must read its own writes stays on the primary.
Outcome
The data tier survives the loss of a member with an automatic election, and replication health is visible on the same dashboards the team already uses.
