Many teams believe their PostgreSQL is highly available because they run a streaming replica. That holds until 3 a.m., when the primary dies and someone has to promote the replica by hand, repoint every application, and work out what to do with the old primary when it comes back. A replica gives you a copy of the data. High availability is everything that happens automatically around that copy.
What a high-availability setup has to do
- Detect failure reliably — without promoting a replica every time the network hiccups.
- Elect exactly one new primary. Two primaries accepting writes at once — split-brain — is far worse than a short outage.
- Move client traffic to the new primary without changing application configuration.
- Bring the old primary back as a replica once it recovers.
- Lose no committed data, or at least a known, agreed amount.
How Patroni does it
Patroni is an agent that runs next to PostgreSQL on every database node and manages it. The nodes coordinate through a distributed configuration store — etcd, Consul, ZooKeeper, or the Kubernetes API. The current primary holds a leader key with a short time-to-live and keeps renewing it. If the key expires, the healthy replicas compete for it, and the most up-to-date one is promoted. A primary that loses contact with the configuration store demotes itself, which is what protects you from split-brain.
The configuration store has to be highly available too: run three (or five) members on separate hosts, ideally in separate availability zones, so it keeps a quorum when one of them fails.
Routing: how applications find the primary
Failover is only useful if clients follow it. Patroni exposes a REST API on every node, and a load balancer such as HAProxy can health-check it: the /primary endpoint returns success only on the current leader, /replica only on healthy replicas. Applications connect to the load balancer, never to a database host directly. A connection pooler such as PgBouncer usually sits in the same path. Alternatives include a floating virtual IP managed by vip-manager, or Kubernetes services when you run on Kubernetes.
Decide how much data you can lose
With asynchronous replication, a failover can lose the last transactions the old primary committed but had not yet shipped. Patroni's maximum_lag_on_failover setting prevents promoting a replica that is too far behind. If you cannot accept any loss of acknowledged commits, synchronous_mode makes every commit wait for a synchronous replica — at the cost of write latency, and with a replica count planned so that one failed node does not stall writes.
Bringing the old primary back
After a failover the old primary's data has diverged and it cannot simply rejoin. Patroni can use pg_rewind to rewind it to the point of divergence and start it as a replica, which is far faster than rebuilding from scratch. pg_rewind requires wal_log_hints = on or data checksums, so enable one of them when you build the cluster, not after the first incident.
The most common gap we find is not in the database layer at all. Patroni promotes a new primary in well under a minute, but the application keeps old connections open or caches DNS, and takes ten minutes to notice. Measure failover from the user's point of view, not from the database log.
Test it before you need it
An untested failover is a hypothesis. Run a planned switchover with patronictl switchover, then the unplanned cases: stop PostgreSQL on the primary, power off the node, cut its network, and take down a member of the configuration store. Measure how long writes are unavailable and whether the applications recover on their own. Repeat after major changes — and periodically in production, during low traffic.
Key takeaways
- A streaming replica is a copy of the data; high availability is the automation around it.
- Patroni with a three-member etcd or Consul cluster gives leader election and split-brain protection.
- Route clients through HAProxy or a virtual IP using Patroni's health checks, never to a fixed host.
- Choose between asynchronous and synchronous replication based on how much data you can afford to lose.
- Test failover end to end, including how long the application takes to recover.