Incident Response

PostgreSQL Disk Full: Why pg_wal Keeps Growing and How to Fix It Safely

The disk is at 98%, pg_wal is huge and somebody is already typing rm. Stop them. What actually fills a PostgreSQL disk, what is safe to free right now, and how to make sure it never happens again.

· 11 min read · AG Data Team

"PostgreSQL disk full" is one of the most common database emergencies, and one of the easiest to make worse. The disk hits 100%, writes fail, and when you look around the biggest directory is pg_wal. Deleting files from it looks like the obvious fix. It is the one thing you must not do. Here is what actually fills the disk, how to free space without damaging the database, and how to stop it from happening again.

What happens when PostgreSQL runs out of disk

It depends on which file system fills up. If it's the one holding pg_wal, PostgreSQL can't write WAL and shuts down. The PostgreSQL documentation describes it plainly: if that file system fills up, "PostgreSQL will do a PANIC shutdown. No committed transactions will be lost, but the database will remain offline until you free some space."

Read that again, because it is the good news: committed data is safe. The outage is about space, not corruption. The data loss stories start later, when someone frees space by deleting the wrong files.

Rule number one: never delete files from pg_wal by hand

WAL files that PostgreSQL still keeps are kept for a reason: crash recovery, archiving or a replica that still needs them. Remove them manually and you can end up with a cluster that won't start, a broken replica or a backup chain with a hole in it. The same goes for pg_xact and anything else inside the data directory. PostgreSQL removes old WAL itself once nothing needs it anymore. Your job is to find out why something still needs it.

Step 1: find what is actually using the space

du -sh /var/lib/postgresql/*/main/* | sort -h | tail
df -h

(Adjust the path to your data directory.) You will usually see one of three pictures: pg_wal is huge, base (the tables themselves) is huge, or the space is outside the data directory altogether.

If pg_wal is huge: the four usual causes

1. A replication slot nobody is reading

A replication slot guarantees that WAL is kept until its consumer has received it. If the consumer is gone (a deleted replica, a CDC tool that was switched off, a forgotten logical replication subscriber), WAL accumulates forever. By default nothing stops it: max_slot_wal_keep_size is -1, which means slots "may retain an unlimited amount of WAL files" (replication settings).

SELECT slot_name, slot_type, active, wal_status,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

An inactive slot holding hundreds of gigabytes is your answer. If you are sure nothing will use it again, drop it with SELECT pg_drop_replication_slot('name'); and PostgreSQL will clean up the WAL at the next checkpoint. If it belongs to a replica you still need, that replica will have to be rebuilt or reconnected later. That is a decision for whoever owns it, so ask before you drop.

2. WAL archiving is failing

When archive_command (or an archive library) fails, PostgreSQL keeps retrying and keeps every segment that hasn't been archived. The docs warn that pg_wal "will continue to fill with WAL segment files until the situation is resolved". Typical reasons: the backup bucket's credentials expired, the archive target is full, or the tool was upgraded and the command changed.

SELECT archived_count, last_archived_wal, last_archived_time,
       failed_count, last_failed_wal, last_failed_time
FROM pg_stat_archiver;

A growing failed_count and an old last_archived_time confirm it. Check the PostgreSQL log for the actual error too: the documentation notes that some failures, such as "command not found", make the archiver restart and are not reported in pg_stat_archiver at all. Fix the command, and the backlog drains by itself. Don't "fix" it by setting archive_command to something that always succeeds without archiving: you will free the disk and silently break point-in-time recovery.

3. A replica that fell far behind

If wal_keep_size is set high, the primary keeps that much WAL for standbys. Setting it to tens of gigabytes "just in case" on a small disk is a common way to fill it.

4. Heavy write bursts and checkpoints

A big data load, a mass UPDATE or an index rebuild can generate WAL faster than checkpoints recycle it, especially with a high max_wal_size. This kind of growth is usually temporary, but on a tight disk temporary is enough.

If the space is elsewhere

Emergency actions that are safe

  1. Grow the volume. On most clouds a disk can be extended online in minutes. That is often the fastest and safest way back.
  2. Free space outside the data directory: old logs, dumps, packages cache. Move rather than delete if you're unsure.
  3. Remove the cause: drop the abandoned slot, fix archiving, cancel the runaway query.
  4. Start PostgreSQL and let it recover. It replays WAL and then removes what it no longer needs at checkpoints.

Many teams keep a "ballast" file of a few gigabytes on the database volume, created with fallocate. When the disk fills, deleting it gives you immediate breathing room to start the server and fix the real cause. It costs a bit of disk and can save an hour of panic.

If PostgreSQL won't start even with free space, or you see errors about missing WAL files, stop experimenting. That's the point to get an experienced DBA involved, and our emergency DBA guide explains what to prepare.

How to make sure it doesn't happen again

Key takeaways

PostgreSQL Disk Full WAL Incident Response
Work with us

PostgreSQL out of disk right now?

Call +371 20943683 or write to info@agdata.lv. A senior DBA responds to critical incidents within 15 minutes, 24/7.

Get in touch More from the blog