Traces

Spans

$result = Telemetry::span('billing.recalculate', function ($span) use ($tenant) {
    $span->setAttribute('tenant.id', $tenant->id);

    return $service->recalculate($tenant);
});

The closure form ends the span for you, records exceptions (exception span event + error status) and rethrows. The manual form:

$span = Telemetry::span('phase.one', attributes: ['shard' => 3]);
$span->addEvent('checkpoint', ['rows' => 5000]);
$span->setStatus(SpanStatus::Ok);
$span->end();

Spans are objects, never looked up by name — two concurrent spans with the same name are simply two spans. Nesting follows the call structure: a span started while another is active becomes its child.

Automatic instrumentation

Request root spans are named METHOD /route/{pattern} by default. Behind catch-all routes, name them yourself with Telemetry::nameRequestsUsing(), override the useless http.route label with resolveRouteUsing() (so route tables and histograms group by the logical route), and add attributes at terminate with enrichRequestsUsing(); see Runtime hooks. An explicit updateName() during the request always survives terminate.

Livewire's update endpoint gets logical naming built in: POST /livewire/update identifies nothing, so the root span is named after the component(s) the request actually touched — POST livewire:{component} for a single-component update, POST livewire:batch when one request updates several. The same value replaces the http.route label (the literal template is preserved as http.route.template), and the root span carries the full component list as a livewire.components attribute. An app-level resolveRouteUsing() override still wins.

Query spans are only recorded inside an active trace — no orphan roots from tinker sessions. The ROOT span additionally carries per-request tallies — db.query.count and db.query.time_ms ("12 queries / 48 ms" at a glance, even when individual query spans are filtered by the noise floor).

Consumer (job) spans carry messaging.wait_time_ms — how long the job sat in the queue between dispatch and the attempt starting — backed by the queue.job.wait_time histogram and a queue.jobs.dispatched counter on the producer side.

Request spans carry session.driver and session.hash — a truncated SHA-256 of the session id (never the id itself; it is an authentication credential). The hash is stable across a visit, so one TraceQL query follows a whole visitor journey: { span.session.hash = "3f2a…" }. Disable with instrument.session.

Request spans carry user.id, user.type (the model: user/admin/reseller) and user.guard (the guard that authenticated) — never name or email. Multi-guard apps stay disambiguated: admin #7 and user #7 are different identities. Filter in TraceQL: { span.user.id = "42" && span.user.type = "admin" }. The login POST itself and logout requests are attributed too (the Login/Logout events are remembered within the request). Disable with instrument.user; enrich (explicit PII opt-in) with Telemetry::resolveUserUsing(fn ($user, ?string $guard) => [...]).

Resource attribution

Request, worker-job and scheduled-task spans carry php.memory.peak_bytes and php.cpu.time_ms — the peak memory and CPU time of THAT unit of work (the process-global peak counter is reset per request/job/task, so long-lived workers report honestly). Matching histograms (http.server.memory.peak, http.server.cpu.time, queue.job.*, schedule.task.duration) give p95 memory/CPU per route, per job — and per custom label dimension. Disable with instrument.resources.

With cboxdk/system-metrics installed, spans additionally carry the process' real OS footprint via a ProcessMetrics tracker around each unit of work: process.memory.rss_peak_bytes (sees non-PHP allocations the PHP allocator misses) and process.cpu.utilization — the same mechanism cboxdk/laravel-queue-metrics uses for per-job metrics.

Every sub-span also carries its own php.cpu.time_ms and php.memory.delta_bytes (allocation delta — may be negative), so the trace waterfall shows WHERE the CPU and memory went, not just the totals. Backdated query spans are excluded (their work already happened when they're recorded).

{ name = "order.payment" } | select(span.php.cpu.time_ms, span.php.memory.delta_bytes)

{ kind = server && span.php.memory.peak_bytes > 134217728 }  # requests over 128 MB

Custom dimensions (context)

Faceted trace search — set the dimensions once, applied everywhere:

// e.g. in middleware, after tenant/team resolution:
Telemetry::context([
    'team.id' => $team->id,
    'team.name' => $team->slug,
    'plan' => $team->plan,
]);

A null value means not set: it removes the dimension rather than recording an empty one, so an optional value needs no filtering at the call site, and a dimension can be cleared without resetContext() (which would drop the trace continuation with it).

Telemetry::context(['tenant.id' => $tenant?->id]); // absent when null
Telemetry::context(['tenant.id' => null]);         // removes it again

From that point every span, event and telemetry-channel log record in the request carries the dimensions (span-specific attributes win on conflict) — and dispatched jobs inherit them, together with messaging.origin.name (the dispatching request/command name), so a job is queryable by team AND traceable back to the exact request that queued it:

{ span.team.name = "checkout" && kind = consumer }
{ span.messaging.origin.name = "POST /demo/orders" }

Context clears automatically between requests and jobs. It also crosses a real HTTP service boundary via the W3C baggage header — see Context propagation below.

Metric dimensions (bounded!)

Context is traces/events/logs only — metric labels multiply cardinality. For bounded dimensions (plan, tier, team — never raw ids) opt in to extra request-duration labels:

Telemetry::labelRequestsUsing(fn ($request) => [
    'plan' => $request->user()?->plan ?? 'guest',
]);

That enables per-plan latency in PromQL:

histogram_quantile(0.95, sum by (le, plan)
  (rate(http_server_request_duration_seconds_bucket[5m])))

Core labels (http.route, method, status) always win over resolver labels; a throwing resolver is reported and ignored.

Context propagation

Outbound propagation uses the full W3C traceparent — trace id and span id — so downstream spans are children, not detached roots:

• Queued jobs: payloads automatically carry the dispatcher's traceparent; workers continue it. (Sync jobs run inline in the dispatcher's context.)
• Incoming HTTP: the middleware continues traceparent headers when traces.continue_incoming is on.
• Outbound HTTP: opt in per request with the client macro (deliberate, so trace headers never leak to third parties by accident):

Http::withTraceparent()->post($url, $payload);

The macro is a no-op when no trace is active.

Telemetry::context() dimensions travel the same way — the macro also attaches a W3C baggage header (team.id=42,plan=pro, percent-encoded) whenever context is set, so a downstream SERVICE inherits the SAME custom dimensions, not just the trace id. The receiving app merges an incoming baggage header back into its own context (instrument.baggage, default on) — gated on traces.continue_incoming too, since baggage is caller-supplied, unvalidated data and should follow the same trust boundary as continuing the trace itself.

Span links (retries)

Not every causal relationship is a parent. A retried job's attempt N+1 is a SIBLING of attempt N — both are children of the original dispatch span, not a continuation of one another — so nesting them as parent/child would misrepresent the shape. Instead (instrument.queue_retry_links, default on), attempt N+1's span carries an OTel span link back to attempt N's span:

{ span.queue.retry = true }

The link is bridged via the app's own cache (queue.retry_link_store/ queue.retry_link_ttl, default 86400s), keyed by the job's stable UUID — a retry can land on a different worker process, so this can't be in-memory state. A null/array cache driver just means retries go unlinked, same graceful degradation as everything else here.

The trace id as a support reference

The trace id doubles as the reference that ties error trackers, support cases and logs back to the trace:

• X-Trace-Id response header on every traced request (traces.response_header, set null to disable).
• Laravel Context: trace_id is added at trace start — Sentry (≥ 4.x), Flare and every log channel pick it up automatically. An explicit Sentry scope tag is set too (traces.share_context).
• Error pages: Telemetry::traceId() is available while the error view renders — show it as “quote this reference id to support”.

The full flows (Sentry → trace, support case → trace, error page recipe) live in Error tracking & support flow.

Sampling

traces.sample_rate (0–1) decides once per trace, at the root. Children inherit the decision; remote callers' decisions are respected via the sampled flag. Unsampled spans still exist as context — ids propagate — but are never buffered or exported.

Error spans escape sampling (traces.always_sample_errors, default on): a 10%-sampled app still exports every failing span. The escaped span's trace may be partial — healthy siblings were dropped under the head decision.

Per-route overrides via the Sample middleware — the re-decision covers the whole active trace, including the still-open request span:

use Cbox\Telemetry\Http\Middleware\Sample;

Route::get('/health', HealthController::class)->middleware(Sample::never());
Route::post('/checkout', ...)->middleware(Sample::always());
Route::get('/feed', ...)->middleware(Sample::rate(0.01));

Tail detail retention

MANY details when it hurts, a lean skeleton when all is well:

TELEMETRY_TRACES_DETAILS=tail
TELEMETRY_TRACES_SLOW_REQUEST_MS=1000
TELEMETRY_TRACES_SLOW_SPAN_MS=100

In tail mode, detail spans (cache operations, queries) are kept only for traces that turned out interesting: an error span anywhere, a request over slow_request_ms, or a single detail span over slow_span_ms (one slow query keeps the WHOLE trace's details). Healthy fast traces ship the skeleton — root span with all its tallies (db.query.count, cache.event.count, resources) — while counters and histograms flow unconditionally.

The decision happens at flush, when the entire trace is in memory — tail-based detail retention without a collector. Buffer-cap force flushes always keep details: a 5000-span request IS interesting.

Bootstrap visibility

When LARAVEL_START is defined (it is, in every standard public/ index.php), the request trace includes a backdated laravel.bootstrap span covering framework boot up to the middleware stack, and the request span carries laravel.bootstrap_ms.

Half-open client spans

A client span is opened on one framework event and closed on another — RequestSending / ResponseReceived, MessageSending / MessageSent. Where the framework hands back a different object than it started with, the two cannot always be paired, and an unpaired span is neither ended nor exported.

The pairing is by identity wherever an identity survives: outgoing HTTP keys on the PSR-7 request, which both Request wrappers share, so it is exact even for concurrent Http::pool() calls to the same URL. Mail has no such key — Laravel sets no Message-ID before dispatching MessageSending, and most transports clone the message — so it falls back to the innermost open send.

Deliberately, nothing guesses by what the call looks like. Matching a failure to a span by method/host/path closes whichever lookalike happens to be open, and two concurrent calls then swap their statuses and durations. A missing span is much the lesser evil than one carrying someone else's outcome, which reads as data.

These paths therefore go unmatched today:

What it costs, and where

The consequence is twofold: that one call is missing from the trace, and while the span sits on the context stack, later work in the same unit of work is parented under it.

ManagesRequestState::flushRequestState() drops the half-open state, and it runs on Octane/NativePHP request and tick boundaries, and at the start of each non-sync queue job (the latter only when instrument.queue is on). Those are the boundaries that bound it.

Under FPM there is nothing to bound — the process ends after the request, so nothing survives to accumulate.

The case with no boundary is a long-running CLI process that is neither Octane nor a queue worker: an artisan command looping over thousands of redirected HTTP calls, or mail sends whose transport keeps throwing, holds one entry per unmatched operation for the life of the command. If you have such a command, give it explicit boundaries — process in chunks and let the work happen in queued jobs rather than inline, which restores the per-job reset.

Buffering

Finished spans buffer in memory and flush at terminate — export latency happens after the response is sent. The buffer is capped (traces.max_buffer, default 5000) and force-flushes when full, so long-running workers and Octane can't grow unbounded.