On 20 October 2026 a power failure took down one availability zone of Google Cloud's London region, europe-west2-b. Hosts, local storage and network hardware in the zone lost power, and the incident lasted roughly six hours. Banks, payment providers and SaaS platforms across Europe felt it. For anyone running databases in the cloud it is a useful case study: not because a zone failed — zones are designed to fail — but because of what happened in the zones that did not.
What happened
According to Google's preliminary incident report, as summarised by IncidentHub:
- At 05:53 UTC the facility hosting europe-west2-b lost utility power. Both the primary and the backup controllers failed to start the generators, so the site ran on UPS batteries.
- At about 06:00 UTC the batteries ran out, and everything in the zone — servers, storage, network aggregation — went offline. Database replicas in the zone became unreachable.
- Automatic failover moved traffic and database primaries into the two remaining zones. That sudden demand saturated their capacity, and Cloud SQL, AlloyDB, Memorystore and Bigtable were all listed among affected services, with "delayed database failover completions".
- Google declared the incident over at 12:10 UTC, with a small number of customers still affected at 14:03. Some downstream services were degraded for hours after that.
Lesson 1: multi-zone is necessary, but not sufficient
Workloads that lived entirely in europe-west2-b simply went down. Those were the obvious losers. The more interesting group is the teams that had done the right thing — a primary in one zone, a standby in another — and still saw minutes or hours of errors. Failover happened, but it happened at the same moment as everyone else's failover, into zones that were suddenly much busier.
A high-availability design is only as good as its assumptions. "If zone B fails, the standby in zone A takes over" quietly assumes that zone A has spare capacity, that the control plane can promote it quickly, and that nothing in the failover path depends on zone B.
Lesson 2: plan capacity for the day everyone fails over
- Run standbys at full size. A standby that is half the size of the primary saves money every day and costs you the outage on the one day it matters.
- Do not count on creating new resources during a regional incident. In this outage new VM creation was impaired for hours. If your recovery plan says "spin up a new replica", it may not work when you need it most. Pre-provisioned, running standbys do.
- Connection storms are real. After a failover every application reconnects at once. A connection pooler such as PgBouncer and sensible retry back-off in the application keep the new primary from being overwhelmed in its first minutes.
Lesson 3: hunt down hidden cross-zone dependencies
Google noted that cross-zone control-plane calls to the failed zone timed out. Your own stack has the same kind of traps:
- A consensus store (etcd, Consul, ZooKeeper) for Patroni or another failover manager with members spread unevenly, so losing one zone loses quorum.
- A single connection pooler, load balancer or bastion host that happens to sit in the failed zone.
- Backups and WAL archives written only to zonal storage.
- Monitoring and alerting hosted in the same zone as the database it watches — so the alarm goes down with the patient.
We describe how to lay out a resilient PostgreSQL cluster, including where to put the consensus store, in our guide to PostgreSQL high availability with Patroni.
A question worth asking your team this week: when did you last fail over production on purpose? If the honest answer is "never" or "during the last outage", you do not yet know how long your failover takes, what breaks during it, or whether the applications reconnect cleanly.
Lesson 4: decide in advance when to leave the region
Google's advice during the incident was to move workloads to another zone or region where feasible. That is only realistic if a cross-region replica, recent backups in another region and a written runbook already exist. Decide ahead of time what your recovery point and recovery time objectives are, who has the authority to trigger a regional failover, and at what point in an incident you make that call. Our first 15 minutes of a database incident article covers the communication side.
Key takeaways
- Zones fail. The real test is whether the surviving zones can absorb everyone's failover at once.
- Keep standbys running at full size; don't rely on creating new resources during an incident.
- Audit consensus stores, poolers, backups and monitoring for hidden dependencies on a single zone.
- Test failover regularly and decide your cross-region trigger before you need it.