Redis Sentinel is the built-in high-availability system for Redis that is not running in Cluster mode. A group of Sentinel processes watches the master and its replicas, agrees when the master is down, promotes a replica and tells clients where the new master is. It works well, but most outages we see with it come from setups that were never tested with a real failure.
What Sentinel does, and what it does not
According to the Sentinel documentation, Sentinel does four jobs: monitoring, notifications, automatic failover and acting as a configuration provider, so clients ask Sentinel for the current master address instead of hard-coding it. It does not shard data or scale writes; for that you need Redis Cluster. And it does not make replication synchronous, which matters a lot further down.
A minimal configuration that is actually robust
# sentinel.conf, identical on three independent hosts
port 26379
sentinel monitor mymaster 10.0.1.10 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster <redis-password>
# start it (either form works)
redis-sentinel /etc/redis/sentinel.conf
redis-server /etc/redis/sentinel.conf --sentinel
- Three Sentinels, on three independent machines. The docs are explicit: you need at least three for a robust deployment, placed on hosts that fail independently. Two Sentinels are never a valid setup, because failover requires a majority.
- Port 26379 is the default and must be reachable between all Sentinels, or they cannot agree and failover never happens.
- The config file must be writable. Sentinel refuses to start without one, and it rewrites the file itself every time its state changes, so in our experience configuration managers that overwrite it can quietly undo a failover's result.
down-after-millisecondsis how long an instance must fail to answer PINGs (or answer with errors) before a Sentinel considers it down.parallel-syncsis how many replicas are repointed to the new master at the same time.- With
requirepasson Redis, Sentinel needssentinel auth-pass, and replicas needmasterauth. Set both on every node, since any of them may become master.
Quorum is not majority
This is the most misunderstood part of Sentinel. The last argument of sentinel monitor is the quorum: the number of Sentinels that must agree the master is unreachable. It is only used to detect the failure.
- One Sentinel not getting replies marks the master SDOWN (subjectively down). That alone never triggers a failover.
- When at least quorum Sentinels report SDOWN, the master becomes ODOWN (objectively down) and a failover is triggered.
- To actually perform it, the Sentinel leading the failover must be authorized by a majority of all Sentinels (or by more, if quorum is larger than the majority). So with 5 Sentinels and quorum 2, two can trigger a failover, but three must authorize it.
The consequence: Sentinel never fails over in a partition that holds only a minority of Sentinels. Each authorized failover gets a unique configuration epoch, so the newest configuration wins when partitions heal. SENTINEL CKQUORUM checks that the current Sentinels can reach both quorum and majority; run it after every change.
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
redis-cli -p 26379 SENTINEL ckquorum mymaster
redis-cli -p 26379 SENTINEL replicas mymaster
redis-cli -p 26379 SENTINEL sentinels mymaster
redis-cli -p 26379 INFO sentinel # sentinel_tilt:0 is what you want
How Sentinel loses acknowledged writes
The documentation is honest about this: because Redis replicates asynchronously, Sentinel cannot guarantee that acknowledged writes survive a failure. The classic case is a network partition that isolates the old master together with some clients. The majority side promotes a replica; the clients stuck with the old master keep writing to it; when the partition heals, the old master becomes a replica of the new one and those writes are discarded.
You cannot remove this window, but you can bound it:
# redis.conf on the master (and on replicas, which may become master)
min-replicas-to-write 1
min-replicas-max-lag 10
With these settings a master stops accepting writes if no replica has acknowledged within 10 seconds. The trade-off is explicit: if all replicas are down, your master refuses writes too. For a cache that is often the wrong choice; for data you care about it is usually the right one. Combined with sensible persistence (see our comparison of RDB and AOF), it turns "some writes lost" into "at most a few seconds of writes lost".
Docker, NAT and hostnames
- Sentinels discover each other through hello messages that announce their own IP and port. With remapped ports (Docker port forwarding, NAT), they announce addresses nobody can reach. Use
sentinel announce-ipandsentinel announce-port, or host networking. - Replicas are discovered from the master's
INFOoutput. If those addresses are not reachable, Sentinel sees no good replicas and can never fail over. The docs' answer for Redis instances in Docker is to map ports 1:1 or use host networking. - To use DNS names instead of IPs, enable
resolve-hostnames, andannounce-hostnamesif you want Sentinel to hand names to clients, for example for TLS certificate matching. Not every client library expects names here.
Operational details that bite later
- Sentinels never forget. Removed Sentinels and dead replicas stay in every Sentinel's state. After stopping one, send
SENTINEL RESET mymasterto each remaining Sentinel, one after another with at least 30 seconds between them, and check withSENTINEL MASTERthat they agree on the count. - TILT mode. Sentinel depends on the system clock. If it sees a negative or unexpectedly large time jump (2 seconds or more), it enters TILT mode, keeps monitoring but does not act, and exits after 30 seconds of normal behaviour. Heavily overloaded VMs and clock jumps can trigger it;
INFO sentinelshowssentinel_tilt. replica-priority 0means a replica is never promoted, useful for a backup or analytics replica in another region. If you set priorities, set them on every node, including the current master.- Test it.
SENTINEL FAILOVER mymasterforces a failover without asking other Sentinels. Run it in a maintenance window and watch what your application does, not just Redis.
Sentinel with large datasets
A failover drill on a 200 MB test instance proves very little about a 60 GB production dataset. What changes:
- Resynchronization after failover. After promotion, the other replicas are repointed to the new master. In our experience they often need a full resync, which on a large dataset means a fork and an RDB transfer per replica.
parallel-syncs 1keeps them from all resyncing at once, at the cost of a longer period with fewer healthy replicas. - Fork pauses versus
down-after-milliseconds. On big instances, the fork behind BGSAVE or AOF rewrite, slow disks or swap can stall replies for a long time. Too short a timeout turns those stalls into needless failovers. Measure real latency spikes on production-sized data and set the timeout above them. - Stale replicas are skipped. A replica disconnected from the master for more than ten times
down-after-milliseconds(plus the time the master has been down) is considered unsuitable for promotion. A replica that was lagging badly before the failure may not be eligible. - Memory headroom. The new master, the resyncing replicas and the fork all need memory at the same time. Budget it per node, not for the steady state.
- When to stop scaling up. If one master can no longer hold the dataset or the write rate, Sentinel is the wrong tool and Redis Cluster is the next step.
Key takeaways
- Run at least three Sentinels on independent hosts with port 26379 open between them.
- Quorum detects the failure; a majority of Sentinels must authorize the failover.
- Asynchronous replication means acknowledged writes can be lost; bound it with
min-replicas-to-writeandmin-replicas-max-lag. - Use Sentinel-aware clients, and fix announced addresses in Docker or NAT setups.
- On large datasets, plan for full resyncs, fork pauses and memory headroom, and test failover on production-sized data.