Query performance

The design goal is that a query costs the size of its answer, not the size of the data. Five mechanisms get there, and each follows from knowing the query shapes in advance. Full reasoning and measurements are in

What happens to a query

Segments are pruned before they are opened. Each manifest carries the segment's time bounds, a per-label value index, and the distinct label sets it contains. A query that cannot match is skipped with no file I/O at all.

Matchers run once per stream, not once per row. Label sets are interned into a per-segment dictionary, so a segment with a million rows across fifty streams evaluates a selector fifty times.

Only surviving rows are decoded. Row selection reads two columns — a timestamp and a u32 stream id — and produces a row-index list. Decoding a record allocates strings and maps; doing that before filtering means a limit=100 query pays for every row it scans.

The predicate is pushed into Parquet. Wide columns — bodies, attribute maps, span events — are never decompressed for rows that will be discarded.

Limits stop the scan. A bounded collector holds limit records rather than every match, so peak memory is set by what was asked for. Once full, its cutoff lets whole segments be skipped: a limit=10 query over a 30-segment store opens at most three.

Counter summaries

A rate or increase over a long window is the one shape the mechanisms above cannot help with: it is not selective. rate(http_requests_total[90d]) matches everything in the window by construction, so there is nothing to prune and nothing to stop early — the query reads every row it covers. That is why a wide dashboard panel used to cost seconds while a limit=100 log search cost milliseconds.

So a segment carries the answer next to it. When a segment is sealed, telemetryd folds each stream's samples once and writes a small folds.bin beside the Parquet file: how many samples, the first and last timestamp and value, the accumulated increase with counter resets already applied, and — since format 2 — the compensated sum of the values, the least and the greatest. Eighty bytes per stream.

A window that wholly contains a segment can then take that stream's total without opening the segment at all. Only the two segments straddling the window's edges are read, plus whatever has not been sealed yet. Measured on a 30-million-row store: a 24-hour window went from 724 ms to 234 ms, a 7-day window from 6.4 s to 555 ms, and 90 days from 7.1 s to 316 ms. Answers agreed with the row-by-row path on every one of 2 456 values, the largest disagreement being 1.07e-15 — floating-point addition in a different order.

Three properties are worth stating, because each was a decision:

It is exact, not approximate. The summary is a fold over the same rows in the same order, not a sample or a sketch. A query answered from summaries and one answered from rows differ only by the order the additions happen in.

It is a shortcut, not a replacement. When enough segments in a window lack a summary that reading them would cost more than the ordinary scan, the store declines and the query takes exactly the path it took before — the same code, unchanged. A store that has never summarised anything is no slower than one that never had the feature.

Existing segments are summarised in the background. Upgrading does not rewrite the store on startup. A maintenance task folds a few segments at a time, so an upgraded instance reaches full speed within the hour while staying answerable throughout. Nothing about the on-disk format changed: folds.bin is additive, and an older telemetryd ignores it.

Which functions a summary answers

A window that wholly contains a segment takes that segment's summary for these, and reads rows only for the segments at its edges and the unsealed tail:

Every one of them reads rows instead, with the same answer it always had, when:

• the expression has a selector outside such a call — max_over_time(x[7d]) / x reads x's rows, so the whole expression is read from rows;
• it contains a subquery or @, which evaluate at times of their own;
• the window is two hours or narrower, too narrow to hold a whole segment;
• the window does not wholly contain a segment, or holds more segments it must read whole than it can afford: four for a single point, shared between the points. A range query of more than four points can afford none, so a chart takes summaries only where every window it draws is made of whole segments and the unsealed tail — rarely, in practice; charts are what the windowed read below is for;
• the window holds a segment without a summary, or with one in the format before gauges were summarised, or with one that fails its checksum — a background task rewrites those, eight every thirty seconds;
• late data puts a stream's samples out of time order across two segments;
• for rate and increase, a NaN in a counter summary — written before staleness markers were left out of summaries;
• for sum_over_time and avg_over_time, a sum that could overflow in some order of adding it up — the count times the largest magnitude, doubled for rounding, past the largest float — or that is not finite. [1e308] then [1e308, -1e308] is infinite in time order and finite segment by segment, and Prometheus's answer is the time-order one. A NaN value makes the sum NaN and is read from rows too; the count, least and greatest of a gauge that held a NaN still come from summaries.

There is no second resolution and no step alignment: a summary is the exact fold of the segment it sits beside, so the answer does not depend on where the query's points fall. Over a sealed week of 850 series, max_over_time and avg_over_time over the whole week — instant queries, one point each — went from 167 ms to 3.5 ms; the summaries cost 2.9 MB beside 38 MB of samples.

Range queries read only where they look

A range query evaluates its expression at every point, and at each one a selector looks only a little way back: five minutes for a bare selector, the range for x[15m]. A week at 250 points is a point every forty minutes, so most of the span is never looked at — and it used to be read anyway, every sample of it held, grouped and sorted. Now only the windows in front of the points are kept, read a series at a time from the store's columns. The answer cannot change, because nothing outside those windows is ever evaluated; an expression with a subquery or @, whose times are its own, still reads its span whole. Over the same week, a bare gauge chart went from 167 ms to 17 ms, an avg_over_time(x[15m]) chart from 164 ms to 15 ms, and the counter-increase panel the UI draws (x - x offset 15m) from 192 ms to 42 ms.

Queries that touch no data at all

/loki/api/v1/labels, /api/v1/label/{name}/values and /series are answered from segment metadata. No Parquet file is opened, and the answer is exact — there is no cardinality cutoff to under-report from.

Trace by id

A trace id has no useful ordering and cardinality equal to the row count, so nothing statistical can prune it. Each segment carries a Bloom filter over its trace ids, which answers "definitely not here" exactly. A lookup reads about one segment instead of every segment in the retention window.

Correctness is layered

Every one of these mechanisms may over-select. None may under-select. The record-level predicate remains the sole authority on what is returned.

That is what makes the optimisations safe to add and safe to remove: getting one subtly wrong costs speed, never correctness. The test suite asserts both halves — that pruning happens, and that the result equals a full scan.

Watching it

curl -s http://127.0.0.1:4319/metrics | grep query_segments

telemetryd_query_segments_scanned_total against telemetryd_query_segments_pruned_total is the ratio to watch when queries feel slow. A high scan count means something is defeating pruning — usually a query with no selective matcher.

telemetryd_query_rows_read_total counts the metric rows queries took out of the store — decoded from segments or read from the buffer. A metric query reads only the series it names: a chart of one app's request counts reads those counts, not the histogram buckets stored beside them. When this grows far faster than the answers queries return, a query is reading series it then throws away.

Where a general engine stays ahead

Named so the trade is explicit: large aggregations over billions of rows, arbitrary GROUP BY and joins, specialised compression codecs, and mature parallel query execution. telemetryd does not compete on any of them, and that is the trade that makes a single binary with no planner and no merge scheduler possible.