Monitoring

Redis Latency Spikes: How to Find the Cause with SLOWLOG, LATENCY and redis-cli

Redis answers in microseconds until suddenly it does not. Timeouts appear, p99 jumps, and the CPU graph looks fine. Here is a practical order of checks that finds the real cause.

· 12 min read · AG Data Team

Most Redis latency spikes have nothing to do with Redis being "slow". Redis executes commands on one main thread, so anything that blocks that thread for 50 milliseconds makes every client wait 50 milliseconds. The job is to find what blocked it: one heavy command, a fork for a snapshot, a disk that stalled an fsync, the hypervisor, or the network between the app and the server.

First: is it Redis or the path to Redis?

Application timeouts measure the whole round trip. Before changing any Redis setting, check where the time goes:

redis-cli -h redis1 --latency            # round-trip latency from this client
redis-cli -h redis1 --latency-history    # the same, in time windows
redis-cli --intrinsic-latency 100        # run ON the server host, 100 seconds
redis-cli -h redis1 INFO | grep latest_fork_usec

The Redis latency guide explains that --intrinsic-latency does not talk to Redis at all. It measures how long the kernel or hypervisor keeps a process off the CPU, so it must run on the server host. If that number is already several milliseconds, no Redis tuning will get you below it. The same guide reports that virtualized hosts with noisy neighbours can be far worse than bare metal.

If --latency from the application host is high but low from the Redis host itself, look at the network, connection churn and the client. Opening a new connection for every request, or making hundreds of round trips instead of using MGET, pipelining or a script, adds latency that Redis never sees.

SLOWLOG: commands that blocked the server

CONFIG GET slowlog-log-slower-than   # default 10000 (microseconds = 10 ms)
CONFIG GET slowlog-max-len           # default 128 entries
SLOWLOG LEN
SLOWLOG GET 25                       # newest entries: id, time, duration in us, args, client
SLOWLOG RESET

According to the SLOWLOG documentation, the slow log measures only command execution time, not network I/O or sending the reply, which is exactly the part where the main thread cannot serve anyone else. Each entry includes the duration in microseconds, the arguments and the client address, so you usually see straight away which service sent it.

Typical findings:

The default threshold is 10 ms and only 128 entries are kept, so on a busy server old entries disappear quickly. Collect it regularly if you want history. In Valkey 8.1 and newer the slow log is part of COMMANDLOG, which also tracks large requests and large replies; the old slowlog-* parameters still work but are deprecated.

Latency monitor: spikes that are not commands

Some stalls never show up in SLOWLOG because they are not command execution. The latency monitor samples internal events such as fork, aof-write, aof-fsync-always, expire-cycle, eviction-cycle and active-defrag-cycle. It is off by default (threshold 0) and can be enabled at runtime:

CONFIG SET latency-monitor-threshold 100   # milliseconds, 0 = off (default)
LATENCY LATEST
LATENCY HISTORY fork
LATENCY DOCTOR

Set the threshold to what your application considers too slow. Each event keeps a time series of 160 samples. LATENCY DOCTOR prints a readable summary with hints. Since Redis 7.0 there is also per-command latency tracking, with p50, p99 and p99.9 exported in INFO latencystats; it is on by default.

The usual suspects outside commands

Last resort: the software watchdog (CONFIG SET watchdog-period 500) logs a stack trace when the server is blocked longer than the period. The documentation calls it experimental, for debugging only. Turn it off again with watchdog-period 0.

What changes with large datasets

A practical order of checks

  1. Measure from both sides: --latency from the app host and from the Redis host, and --intrinsic-latency on the server.
  2. Read SLOWLOG GET and match entries to the time of the timeouts.
  3. Enable the latency monitor and check LATENCY LATEST and LATENCY DOCTOR.
  4. Correlate spikes with snapshots, AOF rewrites, expiry bursts and deploys.
  5. Check the OS: THP, swap, disk latency, CPU steal on VMs.

Key takeaways

Redis Valkey Latency SLOWLOG
Work with us

Redis timeouts in production?

We investigate Redis and Valkey latency on large production instances: slow commands, big keys, fork and persistence settings, and the OS and network underneath.

Get in touch More from the blog