ClickHouse "Too many simultaneous queries" (Error 202): Causes and Fixes

The ClickHouse error "Too many simultaneous queries" (code 202, TOO_MANY_SIMULTANEOUS_QUERIES) means the server refused to start a query because a concurrency limit was already reached. Nothing is broken and no data is at risk. But for the dashboard or service that got the error it looks like an outage, and simply raising the limit is often the wrong fix. The useful questions are which limit fired and what is holding the slots.

The usual form of the message:

Code: 202. DB::Exception: Too many simultaneous queries. Maximum: 1000. (TOO_MANY_SIMULTANEOUS_QUERIES)

Six limits, one error code

In ProcessList.cpp, where ClickHouse registers every new query, the same error code is thrown by six different checks. The wording of the message tells you which one you hit:

Too many simultaneous queries. Maximum: N                      -- max_concurrent_queries (server)
Too many simultaneous select queries. Maximum: N, current: M   -- max_concurrent_select_queries
Too many simultaneous insert queries. Maximum: N, current: M   -- max_concurrent_insert_queries
Too many simultaneous queries for all users. Current: M, maximum: N  -- max_concurrent_queries_for_all_users
Too many simultaneous queries for user alice. Current: M, maximum: N -- max_concurrent_queries_for_user
Too many simultaneous waiting queries. Maximum: N, waiting: M        -- max_waiting_queries

• max_concurrent_queries is the server-wide hard limit, set in the server configuration. The built-in default in ServerSettings.cpp is 0 (unlimited), but the config.xml shipped with the server sets it to 1000. That is why "Maximum: 1000" is the number most people see.
• max_concurrent_select_queries and max_concurrent_insert_queries are separate server-level caps per query kind. Both default to 0, meaning no separate limit.
• max_concurrent_queries_for_all_users is a user-level setting compared against the total number of running queries on the server. It is a soft limit that can differ per profile or user.
• max_concurrent_queries_for_user limits the queries of one user.
• max_waiting_queries is different from the rest: it limits queries waiting for an asynchronous startup job, such as a table still loading after a restart. If you see "waiting queries" right after a restart, the server is still starting up; this is not a load problem.

All the user-level settings above default to 0 (no limit) in the settings reference.

The useful exception: you can still look

The limits do not apply to everything. ProcessList treats KILL QUERY and a SELECT whose first table is system.processes as unlimited queries, along with internal queries. The source comment calls the check "very rough": USE system; SELECT * FROM processes is not recognized, so always write the table name in full. That means that even when every slot is taken you can see what is running and stop it:

-- works even when the server is at its limit
SELECT user, query_kind, is_initial_query, count() AS queries,
       round(max(elapsed)) AS longest_s
FROM system.processes
GROUP BY user, query_kind, is_initial_query
ORDER BY queries DESC;

Columns from system.processes worth knowing: is_initial_query is 0 for child queries, including those sent to shards for distributed execution; the docs warn that for distributed processing queries are sent to remote servers under the default user, so user on a shard may not be the person who started it; initial_user shows who did. On a cluster, a dashboard query fanned out to several shards also shows up as a running query on each server it touches, so the numbers on shards can be higher than on the node clients talk to.

Find the long ones:

SELECT query_id, user, round(elapsed) AS s,
       formatReadableSize(memory_usage) AS mem, left(query, 120) AS q
FROM system.processes
ORDER BY elapsed DESC
LIMIT 20;

And kill what should not be there:

KILL QUERY WHERE query_id = '6e3c1d8a-...';
KILL QUERY WHERE user = 'bi_dashboard' AND elapsed BETWEEN 300 AND 1e9;

What usually fills the slots

• Slow queries piling up. If a query that normally takes 200 ms suddenly takes 20 seconds, the same request rate needs a hundred times more slots. The concurrency limit is where you notice; the cause is whatever made queries slow: a heavy report, a merge storm, memory pressure. Our article on "Memory limit exceeded" in ClickHouse covers the memory side.
• Clients that give up but leave queries running. A BI tool or HTTP client times out after 30 seconds and retries, while the original query keeps running on the server. The setting cancel_http_readonly_queries_on_client_close cancels read-only HTTP queries when the client disconnects; it is false by default (the docs note that the Cloud default is 1). Combine it with a realistic max_execution_time for dashboard users, which is 0, no limit, by default.
• Many tiny inserts. Every INSERT is a query. Thousands of single-row inserts per second eat slots and, worse, create parts; see our guide to "Too many parts" for batching and async inserts.
• Fan-out on a cluster. Distributed queries open child queries on the shards, so a modest number of client queries can look like many more on each node.

To see when rejections happened and for whom, query system.query_log. Queries that failed before they started are logged with type ExceptionBeforeStart:

SELECT toStartOfMinute(event_time) AS minute, user, count() AS rejected
FROM system.query_log
WHERE type = 'ExceptionBeforeStart' AND exception_code = 202
  AND event_time BETWEEN now() - INTERVAL 1 DAY AND now()
GROUP BY minute, user
ORDER BY minute DESC, rejected DESC
LIMIT 50;

Setting the limits properly

The server-wide limit lives in the server configuration, for example in a file in config.d:

# /etc/clickhouse-server/config.d/concurrency.yaml
max_concurrent_queries: 1000
max_concurrent_select_queries: 0   # 0 = no separate limit
max_concurrent_insert_queries: 0

In the current server code these values are re-applied when the configuration is reloaded, so a restart is not needed to change them. Raising max_concurrent_queries is fine if the server has spare CPU and memory and the queries are short. If the slots are full because queries are slow, a higher limit just lets more slow queries compete for the same CPU.

The more useful tool is the soft limit. The source comment for max_concurrent_queries_for_all_users describes exactly this pattern: set it slightly below the hard limit in the default profile, and to 0 (unlimited) for administrators and accounts that are vital for operations, such as metrics exporters. Then, when the server is overloaded, ordinary users are rejected first and you can still connect and investigate. The SQL form below works for profiles and users managed with SQL; if yours are defined in users.xml or users.d, set the same values there:

-- keep headroom: ordinary users stop at 950 concurrent queries server-wide
ALTER SETTINGS PROFILE default SETTINGS max_concurrent_queries_for_all_users = 950;

-- one noisy dashboard cannot take everything
ALTER USER bi_dashboard SETTINGS max_concurrent_queries_for_user = 50;

-- admins and the metrics exporter are not cut off by the soft limit
ALTER USER dba SETTINGS max_concurrent_queries_for_all_users = 0;

queue_max_wait_ms is another option: when the server-wide limit is reached, a query waits up to this many milliseconds for a free slot instead of failing immediately (default 0, no waiting). A short wait smooths out spikes; a long one only hides the problem and holds client connections open.

How to avoid the next one

• Monitor the number of running queries (the Query metric in system.metrics) against your limit, and alert well before it is reached.
• Count code 202 in query_log per user. A handful a day during spikes is a signal; a steady stream means the limits or the workload need attention.
• Give each kind of client its own user with its own limits and max_execution_time, so one dashboard cannot starve ingestion or other services.
• Fix the slow queries. Concurrency problems in ClickHouse are usually latency problems in disguise.