Operate telemetryd on ClickHouse
Operate telemetryd on ClickHouse
Scale out on ClickHouse sets a fleet up. This page is for running one: what each part does when something fails, what to do about it, and the routine work — upgrades, credentials, backups, and watching ClickHouse itself.
One rule underlies all of it: an instance holds no telemetry. A batch is
acknowledged once ClickHouse holds it, and refused with 503 and Retry-After: 30
when it cannot be stored. So nothing an instance does can lose an acknowledged batch,
and what happens to a refused one is the sender's business — see
Where refused batches go.
When ClickHouse stops answering
| What happens | |
|---|---|
| Writes | 503 with Retry-After: 30, for every signal. Nothing is buffered on the instance |
| Queries | 503, with a reason that says ClickHouse is not usable; the server log has ClickHouse's own words |
/ready |
503 not ready, so the load balancer stops sending the instance anything |
/healthz |
ok: the process is fine, and restarting it changes nothing |
/status, telemetryd status |
the failures per operation, and the last error with its time |
| A starting instance | stops, naming the URL, the account and the database; a service unit's Restart=on-failure tries again every five seconds |
| Folds | paused; queries over the hours not folded read the samples, slower but the same answers |
Readiness asks ClickHouse a SELECT 1 with the instance's own account and database,
within two seconds, and reuses the answer for a second. A wrong password or a dropped
database is therefore not ready, where ClickHouse's own /ping would say it was.
When ClickHouse is back, nothing needs doing: instances become ready within a second, senders retry, and the next fold catches up on the hours it missed.
Where refused batches go
What survives an outage is decided by the senders:
- laravel-telemetry without the spool exports at the end of each request and does
not retry: a batch refused with
503is dropped, by design — telemetry never holds up a request. - laravel-telemetry with the spool (
TELEMETRY_OTLP_SPOOL=trueandtelemetry:flush --daemon) keeps refused batches in Redis and retries them, honouringRetry-After. The spool holdsotlp.spool.max_itemsbatches (20,000 by default) and then drops the oldest: size it for the outage you want to ride out, at your batch rate. - An OpenTelemetry SDK or Collector retries
503with backoff for as long as its retry settings say, from memory. - A relay in front of the load balancer — a telemetryd without
[clickhouse]— keeps what it accepted on its own disk and forwards it when the fleet answers again, up tostorage.disk_budget. It is the only one of these that survives a long outage without memory to spare.
One replica down
Each instance talks to the replica in its clickhouse.url. When that replica stops
answering, those instances are not ready and the load balancer sends their traffic to
the others; the replica catches up from Keeper when it returns. Point instances at
different replicas — or at a TCP load balancer over all of them — so that losing one
replica never takes every instance with it.
On several shards, reads and writes across shards go to each shard's first replica that answers, so a replica down costs the shard nothing while another replica of it is up. A whole shard down refuses writes that have rows for it and queries that need it: logs are spread over every shard, so in practice all of them.
Keeper down
Without Keeper, replicated tables are read-only. Reads keep working; an insert waits
for Keeper until clickhouse.timeout (30 seconds) and is then refused with 503 and
Retry-After. Readiness only reads, so instances stay ready: the load balancer
keeps sending them writes, and each is refused. A sender whose own timeout is shorter —
laravel-telemetry's is three seconds — gives up first and treats it as a failed export. Watch
telemetryd_clickhouse_requests_total{operation="insert", outcome="error"}
(alerts), not /ready, for this one.
Folds pause, since their lease lives in a replicated table. An instance restarted on
an unchanged database still starts, since it runs no DDL; a first start, or one with
something to create, waits on ON CLUSTER statements that do not finish without
Keeper. Restarting fixes nothing here: when Keeper is back, inserts succeed again
within seconds.
Run three Keeper nodes, or five: Keeper needs a majority to answer.
A disk filling
ClickHouse refuses inserts on a full disk, so writes are 503 as above. Shorten
retention to free space: change retention.* in the configuration of every
instance (they share the tables, and the last one to start sets the TTL), and send one
of them SIGHUP. The new TTL is applied without rewriting stored parts, and ClickHouse
drops whole day partitions past it — but on its own schedule, which can take hours
(merge_with_ttl_timeout). To free space now, drop the oldest days yourself:
SELECT table, partition, formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts WHERE database = 'telemetry' AND active
GROUP BY table, partition ORDER BY partition LIMIT 20;
ALTER TABLE telemetry.otel_logs DROP PARTITION '2026-07-01'; -- ON CLUSTER telemetry, on a cluster
Every table but telemetryd_meta is partitioned by day. Nearly all the bytes are in
otel_logs, otel_traces and telemetryd_samples, so drop days from those; on
several shards, from their _local tables. The rollups and folds of a dropped day stay
until their own TTL, so log volume charts and long rates still show it while its lines
and spans are gone.
With a hot and a cold volume, also check move_factor: when the hot volume is fuller
than it, ClickHouse moves the oldest parts to the cold one whatever cold_after says.
Too many parts
ClickHouse refuses an insert into a partition with more active parts than
parts_to_throw_insert (3,000 by default), and slows inserts down from
parts_to_delay_insert. telemetryd inserts with async_insert, which gathers many
small batches into few parts, so this means merges are not keeping up, not that
batches are too small:
SELECT table, partition, count() AS parts
FROM system.parts WHERE database = 'telemetry' AND active
GROUP BY table, partition ORDER BY parts DESC LIMIT 10;
SELECT table, elapsed, progress, num_parts FROM system.merges;
The usual causes are a disk that is too slow for the write rate, too few background
merge threads (background_pool_size), or something else writing to the tables in
small inserts without async_insert.
A slow query
Every read telemetryd makes runs with max_execution_time set to clickhouse.timeout
(30 seconds by default), so ClickHouse stops a query telemetryd has given up on, and a
client that disconnects cancels its query. The expensive shapes are the ones
How queries stay fast names; to
find which ones you run:
SELECT event_time, query_duration_ms, read_rows, formatReadableSize(read_bytes), query
FROM system.query_log
WHERE type = 'QueryFinish' AND user = 'telemetryd' AND event_time > now() - INTERVAL 1 HOUR
ORDER BY query_duration_ms DESC LIMIT 10;
Upgrading telemetryd
- Replace instances one at a time. A new version's startup adds what it needs —
tables, views, columns, indexes — and changes nothing an older instance uses, so old
and new run side by side while you roll. Wait for
/readybefore the next one. - Then run
telemetryd clickhouse migrate, once every instance runs the new version: start with--dry-run. Not before: an older instance that restarts creates what its version expects, including objectsmigratehas just dropped. - Read the release notes for anything that needs
--backfillor--materialize-indexes; both read old data once, and can run while the fleet serves.
To roll back, replace instances with the older version the same way. Before migrate
that is always safe; after it, the release notes say whether the older version reads
the newer layout.
Upgrading ClickHouse
Upgrade one replica at a time, as ClickHouse documents, staying within a release line
telemetryd is tested on (25.8 LTS today). Instances pointed at the replica being
upgraded are not ready while it is down. Run telemetryd clickhouse check against the
upgraded server before moving to the next.
Rotating the password
The connection is made at startup, so a new password takes a restart. Rotate without an outage through a second account:
- Create
telemetryd_2with the same grants astelemetryd(Access). - Set
clickhouse.userandclickhouse.passwordto it on each instance, and restart them one at a time. - Drop the old account once
system.processesshows nothing from it.
A SIGHUP with a changed [clickhouse] is refused, naming each setting that needs a
restart; the running instance keeps its connection.
Back up and restore
The instances hold nothing a backup needs: back up the database, including
telemetryd_meta, which records the TTLs applied, from when each rollup is complete,
the shard count and migrate's progress.
BACKUP DATABASE telemetry ON CLUSTER telemetry TO S3('https://…/telemetry/2026-10-25', '…', '…');
RESTORE DATABASE telemetry ON CLUSTER telemetry FROM S3('https://…/telemetry/2026-10-25', '…', '…');
On a cluster, ON CLUSTER backs up every shard once and restores each shard's rows to
the shard they came from. Restore into a cluster with the same number of shards: a row
is on the shard its key picks out of that many, and telemetryd refuses to start on a
database laid out for another count.
The replicated tables' Keeper paths name the database
(/clickhouse/tables/{shard}/{database}/{table}), so two databases of one name cannot
share a Keeper. To restore over the live one, drop it with DROP DATABASE telemetry ON CLUSTER telemetry SYNC first, which removes its Keeper entries. Test restores on a
separate cluster with its own Keeper, and point one instance at it: telemetryd clickhouse check, then a few queries you know the answers to.
A backup is a copy of up to ninety days: at sixty billion lines a month, a few
terabytes. ClickHouse's incremental backups (SETTINGS base_backup = …) copy only parts
that are new since the last one, and old days never change.
Setting up the cluster
telemetryd needs, on every ClickHouse server: a cluster in remote_servers, {shard}
and {replica} macros, Keeper, and on several shards internal_replication. A two-shard
cluster of two replicas each:
<clickhouse>
<zookeeper>
<node><host>keeper-1</host><port>9181</port></node>
<node><host>keeper-2</host><port>9181</port></node>
<node><host>keeper-3</host><port>9181</port></node>
</zookeeper>
<remote_servers>
<telemetry>
<shard>
<internal_replication>true</internal_replication>
<replica><host>ch-1a</host><port>9000</port></replica>
<replica><host>ch-1b</host><port>9000</port></replica>
</shard>
<shard>
<internal_replication>true</internal_replication>
<replica><host>ch-2a</host><port>9000</port></replica>
<replica><host>ch-2b</host><port>9000</port></replica>
</shard>
</telemetry>
</remote_servers>
<!-- different on each server -->
<macros><shard>01</shard><replica>ch-1a</replica></macros>
</clickhouse>
A cluster of one shard is the same with one <shard>. The servers talk to each other
on the native port as the cluster's <user> (default when unset): give that account
a password, or <secret> the cluster, before the servers are reachable from anywhere
else. scripts/clickhouse/ in the repository has the cluster the test suite runs on, as
Docker Compose files.
Watching ClickHouse itself
telemetryd's alerts catch what a sender would
notice: inserts failing or slow. Before that, ClickHouse shows the trouble coming. Its
built-in Prometheus endpoint (<prometheus> in its server config) exports what these
need; the same numbers are in its system tables:
| Watch | Where | Trouble when |
|---|---|---|
| Free disk | system.disks |
under what cold_after days or a day of growth needs |
| Parts per partition | system.parts |
over a few hundred, and rising |
| Replication queue | system.replicas (queue_size, absolute_delay) |
a delay of minutes that keeps growing |
| Read-only replicas | system.replicas (is_readonly) |
any — usually Keeper |
| Keeper | system.zookeeper_connection |
no connection |
| Failed distributed DDL | system.distributed_ddl_queue |
entries that do not finish |
telemetryd_app_bytes_estimate on /metrics shows what each app's telemetry costs on
disk, read from system.parts at most once a minute.