Diagnose a query that returns nothing

An empty result has exactly four causes. Check them in this order — it goes from most to least common.

1. Nothing was stored

telemetryd status

If records are zero for that signal, the problem is on the emitting side. Nothing about the query will fix it.

2. Something was refused

curl -s localhost:4319/metrics | grep telemetryd_ingest_rejected_total

Every rejection is counted with a reason. body_too_large, too_many_labels, invalid_metric_name, missing_trace_id each point somewhere specific. telemetryd never drops data quietly, so a zero here genuinely means nothing was refused.

series_limit is the one that looks least like a problem and is the most confusing to run into. It refuses only new series, so everything already flowing keeps flowing and one signal simply stops appearing — usually logs, because metrics got there first and filled the budget. telemetryd status puts the usage next to the limit:

  series       98432 of 100000 (98.4%)
               98432 of 100000 (98.4%) for checkout, the largest

Past 90% it also says so under Attention.

Series that stop being written give their slots back after limits.series_idle_after (an hour by default), so a full budget means it is full of series that are currently active. Raise limits.max_series — about 1 KB of memory per series — or send fewer distinct label combinations. The usual cause is a label carrying something unbounded: a request id, a user id, a full URL with its query string.

If you would rather never turn a new series away, limits.when_series_full = "evict_oldest" makes the budget a ring buffer instead. Busy series then lose samples rather than new ones being refused; telemetryd_series_evicted_total counts what that costs. The configuration reference has the trade-off in full.

2b. A query was refused

A 400 from a dashboard panel is now recorded on the server too:

sudo journalctl -u telemetryd | grep "refused a query"
curl -s localhost:4319/metrics | grep telemetryd_query_rejected_total

The log line carries the expression and the reason; the counter is labelled by surface (promql, logql) and reason. Before this, a refusal existed only in the client's response — an operator looking at their own instance could not tell which query had failed or that any had.

3. The timestamps are wrong

curl -s localhost:4319/metrics | grep timestamps_rescaled

The most common integration bug is sending seconds or milliseconds where OTLP specifies nanoseconds. telemetryd detects and corrects it — the counter above tells you it is happening — but a producer sending something outside any plausible range gets rejected, and the records land nowhere.

Check what time range you actually have. /status is a single line of JSON, so pick the fields out rather than grepping around them:

curl -s localhost:4319/status |
  python3 -c 'import json,sys,datetime
d = json.load(sys.stdin)["storage"]["logs"]
for k in ("oldest_record_nanos", "newest_record_nanos"):
    v = d[k]
    print(k, datetime.datetime.fromtimestamp(v / 1e9).isoformat() if v else "none")'

none for both means the signal is empty and you are back at step 1. A range that does not overlap the window you queried is the answer — and a range starting in 1970 or in the far future is a producer sending the wrong unit, which is step 3.

4. The query does not match

Work backwards from the least selective query:

# Does anything exist at all?
curl -G localhost:4319/loki/api/v1/query_range \
  --data-urlencode 'query={service_name=~".+"}'

# What labels are there?
curl -s localhost:4319/loki/api/v1/labels

# What values does one have?
curl -s localhost:4319/loki/api/v1/label/app/values

Label discovery is answered from metadata, so it is fast and exact — if a value is not listed, nothing carries it.

Two things that surprise people:

• Label matchers are fully anchored. {job=~"api"} does not match api-gateway. Write api.*.
• Line filters are not anchored. |~ "err" matches anywhere in the line.

Not an empty result: a subset boundary

If the query used something telemetryd does not implement, the response is HTTP 400 with the feature named — not an empty result:

{
  "error": {
    "code": "unsupported_feature",
    "feature": "PromQL function `quantile`",
    "docs": "https://github.com/cboxdk/telemetryd/blob/main/COMPATIBILITY.md"
  }
}

Read the body before concluding there is no data.

Retention already took it

curl -s localhost:4319/status | python3 -m json.tool | grep -A5 retention

deleted_by_budget above zero means the disk budget deleted data still inside its retention window. See sizing the disk budget.