PostgreSQL "terminated by signal 9: Killed": The OOM Killer and What to Change

PostgreSQL "terminated by signal 9: Killed" in the server log almost always means the Linux kernel ran out of memory and its OOM killer picked a postgres process. PostgreSQL did not crash on its own; something outside it sent SIGKILL. What makes this incident painful is what comes next: one dead backend forces the postmaster to throw out every other session and run crash recovery.

What the logs look like

On the server side you will see a sequence like this. On PostgreSQL 17 and older the first line says server process; PostgreSQL 18 names the process type instead, for example client backend (PID ...).

LOG:  server process (PID 48213) was terminated by signal 9: Killed
DETAIL:  Failed process was running: SELECT c.region, count(DISTINCT o.customer_id) ...
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at ...
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at ...
LOG:  database system is ready to accept connections

The DETAIL: Failed process was running line is the most useful one in the whole incident: it shows the query that was executing in the killed process. Applications see a different message on every session that was open at that moment:

FATAL:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current
         transaction and exit, because another server process exited abnormally and
         possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.

For the next seconds or minutes, new connections are refused with FATAL: the database system is in recovery mode. If you need to go deeper into the startup phase and the "redo in progress" lines, we covered that in PostgreSQL "Connection refused".

Why one killed query takes everything down

PostgreSQL backends share memory: shared_buffers, lock tables, WAL buffers. A process killed with signal 9 gets no chance to release locks or finish writing to that shared area, so the postmaster cannot trust it any more. It tells all other processes to exit, reinitializes shared memory and replays WAL from the last checkpoint. That is deliberate. It is the price of not risking corrupted data.

Whether the server restarts automatically depends on restart_after_crash, which is on by default. The documentation notes it can be useful to turn it off when PostgreSQL runs under clusterware, so the HA tool decides what to do instead.

If the OOM killer picks the postmaster itself, there is no automatic recovery: the instance stays down until it is started again. That is why the docs recommend protecting the postmaster (see below).

Step 1: prove it was the OOM killer

Signal 9 can also come from a person running kill -9, a watchdog or an orchestrator. Check the kernel log before you change any settings:

# who sent signal 9? the kernel log says if it was the OOM killer
journalctl -k --since "2 hours ago" | grep -i -E "out of memory|oom"
dmesg -T | grep -i -E "killed process|oom"

# is postgres running inside a memory-limited cgroup (systemd, docker, k8s)?
systemctl show postgresql -p MemoryMax -p MemoryHigh
cat /sys/fs/cgroup/$(cut -d: -f3 /proc/$(pgrep -o postgres)/cgroup)/memory.max

# current overcommit mode and OOM protection of the postmaster
sysctl vm.overcommit_memory vm.overcommit_ratio
cat /proc/$(pgrep -o postgres)/oom_score_adj

The PostgreSQL documentation shows the kernel message as Out of Memory: Killed process 12345 (postgres); the exact wording depends on the kernel version. If the kill happened inside a cgroup (a systemd MemoryMax, a Docker or Kubernetes memory limit), the kernel log will mention the memory cgroup. That changes the fix: the host may have plenty of free RAM, and the limit you are hitting is the container's.

Step 2: find what used the memory

• The query in Failed process was running. Run EXPLAIN on it and look for several sorts and hash operations in one plan, especially with parallel workers.
• work_mem multiplied out. The documentation is explicit: work_mem (default 4MB) is a limit per sort or hash operation, not per query. A complex query may run several at once, many sessions may do so concurrently, and each parallel worker gets its own limit. Hash operations may use work_mem × hash_mem_multiplier, and the default multiplier is 2.0.
• Connections. Every backend has its own private memory on top of shared memory. Hundreds of active backends plus a generous work_mem is the classic recipe.
• Maintenance work. Index builds and other maintenance operations use maintenance_work_mem per process. Several of them at once on top of normal traffic can push a server over the edge.
• Other processes on the host. Backup agents, ETL jobs or a JVM sharing the machine often trigger the OOM killer, and the kernel still picks a postgres process to kill.

While a suspicious session is still running, a superuser can call pg_log_backend_memory_contexts(pid) (PostgreSQL 14+). It writes that backend's memory contexts to the server log, so you can see which part of the executor is growing. Your own session's contexts are visible in the pg_backend_memory_contexts view.

Step 3: fix it so it does not repeat

1. Budget memory honestly. Count shared_buffers, plus realistic concurrency × operations per query × work_mem, plus maintenance jobs, plus the OS and everything else on the host. If the sum does not fit, lower work_mem globally and raise it only for the roles or sessions that need it (SET work_mem in a reporting job).
2. Cut the number of backends. The documentation itself suggests reducing max_connections and using a connection pooler instead. A transaction pooler in front of PostgreSQL caps how many backends exist at once; see our PgBouncer guide.
3. Switch to strict overcommit. With vm.overcommit_memory=2 the kernel refuses allocations beyond its commit limit, so PostgreSQL gets an ordinary ERROR: out of memory in one query instead of a SIGKILL that resets every session. The docs say this does not prevent the OOM killer entirely but lowers the chances significantly. Be careful with the limit: per the kernel documentation, in mode 2 the commit limit is swap plus vm.overcommit_ratio of physical RAM, and that ratio defaults to 50%. On a big server with little swap that can be far less than you expect, so set vm.overcommit_ratio (or vm.overcommit_kbytes) deliberately and test under load before relying on it.
4. Protect the postmaster. The documented approach is to start the postmaster with an OOM score adjustment of -1000 and let child processes reset to 0:

   # root-owned startup script / systemd unit, before starting postgres
   echo -1000 > /proc/self/oom_score_adj
   export PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
   export PG_OOM_ADJUST_VALUE=0

   This does not stop backends from being killed, but it keeps the supervisor alive, so the instance recovers on its own instead of staying down. Some distribution packages already configure this in their systemd units; check before you add it.
5. Fix the container limit, not only PostgreSQL. If the cgroup limit was hit, either raise it or size shared_buffers and work_mem against the limit, not against the host's RAM.

On large databases this hurts more

• Recovery time grows with WAL. After the reset, PostgreSQL replays everything written since the last checkpoint. Servers tuned for heavy writes often have a large max_wal_size and a long checkpoint_timeout, so replay can take minutes, not seconds. Measure it on a full-size copy, so your failover and health-check timeouts match reality.
• The cache starts cold. Shared memory is recreated, so shared_buffers is empty after the restart. On a multi-hundred-GB working set the first period after recovery is slower than normal, and a retry storm from applications at that moment can make it worse.
• Big analytical queries are the usual trigger. Hash joins and aggregates over billions of rows, run with several parallel workers each, multiply work_mem quickly. A work_mem that was fine when the tables were small can become dangerous as data grows, with no configuration change at all.
• HA has to agree with the restart policy. With Patroni or similar tools, decide whether a local crash restart or a failover is the desired outcome, and set restart_after_crash and the tool's timeouts accordingly. Repeated OOM kills on the primary are a reason to fix memory, not to fail over again and again.