Incident Response

Redis MISCONF Error: "Unable to Persist to Disk" and How to Fix It

Every write suddenly fails with MISCONF, and so does PING. Redis has not crashed: its last background snapshot failed and it is refusing writes on purpose. Here is how to find out why the save failed and how to get writes back without throwing away the safety net.

· 16 min read · AG Data Team

The Redis MISCONF error tends to show up at the worst moment: the application log fills with failed writes, health checks go red, and yet the Redis process is alive and answering reads. That last detail is the clue. MISCONF is not a crash. Redis tried to save a snapshot, the save failed, and since then it has been rejecting writes on purpose so that you notice.

This is the full text clients receive:

(error) MISCONF Redis is configured to save RDB snapshots, but it's currently
unable to persist to disk. Commands that may modify the data set are disabled,
because this instance is configured to report errors during writes if RDB
snapshotting fails (stop-writes-on-bgsave-error option). Please check the
Redis logs for details about the RDB error.

Valkey returns the same message with its own name in it, and everything below applies to it as well.

Why Redis stops accepting writes

The behaviour comes from one option, stop-writes-on-bgsave-error, which is yes by default. The comment in redis.conf explains the reasoning: if snapshots are enabled and the latest background save failed, Redis makes the problem visible "in a hard way", because otherwise nobody would notice that data is no longer reaching the disk until a restart wiped it out.

The check in server.c is short. Writes are refused when all three are true:

Two practical consequences are easy to miss:

A related message, MISCONF Errors writing to the AOF file: ..., comes from the append-only file. It is not controlled by stop-writes-on-bgsave-error: when AOF is on and a write or fsync to it fails, Redis refuses writes until the AOF can be written again. The causes are usually the same disk problems described below.

Step 1: confirm the state

redis-cli INFO persistence | grep -E 'rdb_(last_bgsave_status|bgsave_in_progress|changes_since_last_save|last_save_time|last_cow_size)'

redis-cli CONFIG GET save
redis-cli CONFIG GET dir
redis-cli CONFIG GET dbfilename

The fields from INFO persistence that matter here:

CONFIG GET dir and dbfilename show where Redis is trying to write. Write those values down; you will need them for the disk checks, and they are also worth a security look (see the callout below).

Step 2: read the log, it names the cause

The error text tells you to check the logs, and that is good advice: the actual reason is printed there, not in the reply to the client. The messages come from rdb.c. These are the ones you will see most often:

Background saving started by pid 48211
Write error while saving DB to the disk(rdbSaveRio): No space left on device
Background saving error

Failed opening the temp RDB file temp-48211.rdb (in server root dir /var/lib/redis) for saving: Permission denied

Can't save in background: fork: Cannot allocate memory

Background saving terminated by signal 9

"No space left on device": the disk is full

The most common cause. Redis writes the new snapshot to a temporary file named temp-<pid>.rdb in dir and only then renames it over the old dump. So for a moment you need room for the old dump and the new one side by side. A volume that holds a 6 GB dump with 4 GB free will fail every save, even though it does not look full. Failed saves also leave nothing useful behind: after a write error Redis deletes the temp file, so the space it briefly used is not obvious afterwards.

Look for whatever else shares the volume: AOF files, application logs, old copies of dump.rdb that someone made "just in case". Free space, or move dir to a larger volume in the config and restart in a maintenance window.

"Permission denied" or "Read-only file system"

Typical after a migration, a restore from backup with the wrong owner, a container volume mounted with different UID, or a filesystem that the kernel remounted read-only after I/O errors. Compare the owner of dir with the user Redis runs as. If the filesystem went read-only on its own, treat it as a disk failure first and a Redis problem second.

"fork: Cannot allocate memory"

BGSAVE forks a child process that writes the snapshot while the parent keeps serving clients. With the default Linux setting vm.overcommit_memory = 0, the kernel rejects requests it considers obvious overcommits, so forking a large process can fail when free RAM plus swap looks too small, even though the child, thanks to copy-on-write, usually needs far less. Redis knows this: at startup it checks the setting and logs that memory overcommit must be enabled, otherwise "a background save or replication may fail under low memory condition". The fix it suggests is vm.overcommit_memory = 1 in /etc/sysctl.conf and sysctl vm.overcommit_memory=1 to apply it now (see syscheck.c).

"terminated by signal 9": the child was killed

The fork succeeded, but the child was killed before it finished, very often by the kernel OOM killer. While the child writes, every page the parent modifies is copied, so a write-heavy instance needs real headroom above used_memory during a save. dmesg will show the kill. The long-term fixes are to leave more RAM free, to save less often during write peaks, or to bring the dataset down with a sensible maxmemory; our guide to Redis OOM and eviction policies covers the sizing side.

The OS checks for all four causes:

df -h /var/lib/redis            # free space where the dump is written
ls -ld /var/lib/redis           # owner and mode vs the user Redis runs as
sysctl vm.overcommit_memory     # Redis warns at startup unless this is 1
dmesg -T | grep -i -E 'oom|killed process'

Step 3: get writes back

Once the cause is fixed, trigger a save yourself instead of waiting for the next save point:

redis-cli BGSAVE
redis-cli INFO persistence | grep -E 'rdb_bgsave_in_progress|rdb_last_bgsave_status'
# rdb_last_bgsave_status:ok  -  writes are accepted again

When the status turns ok, writes resume with no restart. If it fails again, the log line will tell you what is still wrong.

If the business cannot wait while you free disk space or add memory, there is an escape hatch:

redis-cli CONFIG SET stop-writes-on-bgsave-error no
# ...fix the disk / memory problem, run BGSAVE, confirm status ok...
redis-cli CONFIG SET stop-writes-on-bgsave-error yes

With the option off, Redis accepts writes even though nothing is being persisted. That is a conscious trade: the application works again, but every write since the last good snapshot lives only in RAM. Use it as a bridge for minutes or hours, keep the incident open, and switch the option back on once a save succeeds. Do not "fix" MISCONF by putting stop-writes-on-bgsave-error no into the config file and forgetting about it; the comment in redis.conf suggests that only when you have proper monitoring of persistence in place.

Two situations where the right answer is different:

Check where dir points. If CONFIG GET dir returns something like /root/.ssh, /etc or a cron directory, and dbfilename is not a dump file name, this is not a disk problem. Redis instances exposed to the network without authentication are a known target: an attacker changes dir and dbfilename at runtime and uses a save to drop a file onto the host. A failing save in an odd location can be the visible trace of that. Since Redis 7.0, dir and dbfilename are protected configs and cannot be changed with CONFIG SET unless enable-protected-configs allows it. Treat such a finding as a security incident.

How to avoid the next one

Key takeaways

Redis Valkey Persistence Troubleshooting
Work with us

Redis refusing writes right now?

We can get on your Redis or Valkey servers quickly, find out why persistence failed and bring writes back without losing data. Tell us what you are seeing.

Get in touch More from the blog