Traces

Spans

$result = Telemetry::span('billing.recalculate', function ($span) use ($tenant) {
    $span->setAttribute('tenant.id', $tenant->id);

    return $service->recalculate($tenant);
});

The closure form ends the span for you, records exceptions (exception span event + error status) and rethrows. The manual form:

$span = Telemetry::span('phase.one', attributes: ['shard' => 3]);
$span->addEvent('checkpoint', ['rows' => 5000]);
$span->setStatus(SpanStatus::Ok);
$span->end();

Spans are objects, never looked up by name — two concurrent spans with the same name are simply two spans. Nesting follows the call structure: a span started while another is active becomes its child.

Automatic instrumentation

Request root spans are named METHOD /route/{pattern} by default. Behind catch-all routes, name them yourself with Telemetry::nameRequestsUsing(), override the useless http.route label with resolveRouteUsing() (so route tables and histograms group by the logical route), and add attributes at terminate with enrichRequestsUsing(); see Runtime hooks. An explicit updateName() during the request always survives terminate.

Livewire's update endpoint gets logical naming built in: POST /livewire/update identifies nothing, so the root span is named after the component(s) the request actually touched — POST livewire:{component} for a single-component update, POST livewire:batch when one request updates several. The same value replaces the http.route label (the literal template is preserved as http.route.template), and the root span carries the full component list as a livewire.components attribute. An app-level resolveRouteUsing() override still wins.

Query spans are only recorded inside an active trace — no orphan roots from tinker sessions. The ROOT span additionally carries per-request tallies — db.query.count and db.query.time_ms ("12 queries / 48 ms" at a glance, even when individual query spans are filtered by the noise floor).

Consumer (job) spans carry messaging.wait_time_ms — how long the job sat in the queue between dispatch and the attempt starting — backed by the queue.job.wait_time histogram and a queue.jobs.dispatched counter on the producer side.

Request spans carry session.driver and session.hash — a truncated SHA-256 of the session id (never the id itself; it is an authentication credential). The hash is stable across a visit, so one TraceQL query follows a whole visitor journey: { span.session.hash = "3f2a…" }. Disable with instrument.session.

Request spans carry user.id, user.type (the model: user/admin/reseller) and user.guard (the guard that authenticated) — never name or email. Multi-guard apps stay disambiguated: admin #7 and user #7 are different identities. Filter in TraceQL: { span.user.id = "42" && span.user.type = "admin" }. The login POST itself and logout requests are attributed too (the Login/Logout events are remembered within the request). Disable with instrument.user; enrich (explicit PII opt-in) with Telemetry::resolveUserUsing(fn ($user, ?string $guard) => [...]).

Query diagnostics

Every db.query span carries laravel.db.query.shape: a 16-hex hash of the statement with its values taken out (string and numeric literals become ?, any in (…) list becomes in (?), multi-row values collapse to one row), scoped to the connection. Group by it to see one row per query kind, however many ids or list lengths it ran with.

Repeated queries within one trace are flagged as one of two smells (instrument.query_duplicates, default on; threshold instrument.query_duplicates_threshold, default 3):

Each event fires once per distinct query, at the threshold crossing, and carries the SQL text, laravel.db.query.shape and db.query.repeat_count. Query spans from the crossing onward are marked laravel.db.query.repeat = "n_plus_one" | "duplicate"; treat every span sharing a flagged shape as part of the group. A loop of writes is not called N+1 — it is a missing bulk insert, a different fix.

Where it came from. The first stack frame outside vendor/ and this package is attached as code.file.path (relative to the app), code.line.number and code.function.name — the $order->items in the Blade loop, not Eloquent's internals. instrument.query_origin decides which spans pay for the backtrace:

• flagged (default) — slow queries (≥ traces.details.slow_span_ms) and the crossing occurrence of a repeated one, plus the N+1/duplicate events.
• all — every query span (a backtrace per query; local debugging).
• off — never.

With traces.details = tail, a trace with a repeated query keeps its detail spans — the trace that shows the loop is the one worth keeping, even when every query in it was fast.

{ span.laravel.db.query.repeat = "n_plus_one" }
{ span.db.query.n_plus_one.count > 0 && kind = server }

Root spans also count slow queries (≥ traces.details.slow_span_ms) as db.query.slow.count, so "which traces have a slow query" is one search.

N+1 outgoing calls

The same loop that runs a query per row can make an API call per row: GET /users/17, GET /users/42, … one after another. Outgoing client spans are grouped by shape the way queries are (instrument.http_client_n_plus_one, default on; threshold instrument.http_client_n_plus_one_threshold, default 3):

• The shape is the method, the host (and port), the path with id-like segments replaced by {id}, and the sorted query parameter names without their values: GET api.example.com/users/{id}?include. An id is a number or date, a UUID, a ULID, a hex digest (16+ chars), a Stripe-style token (cus_NffrFeUfNV2Hib), a long letters-and-digits token or an email address; v2, oauth2 and other names stay as they are, and /orders/17.json is /orders/{id}.json. Every client span carries laravel.http.client.shape (16 hex) and url.template (/users/{id}).
• N+1 is one shape called with different URLs, one after another, threshold times in one trace. Reads only (GET/HEAD) — a loop of POSTs is a missing bulk endpoint, not N+1. The same URL again is a retry or a missing cache, not a per-row fetch. Calls sent together through Http::pool are the fix, so a call that starts while another of its shape is in flight is not counted. Pagination is not N+1: a paging query parameter (page, cursor, offset, after/before, starting_after, page_token/pageToken, since_id/max_id, page[…], …) or a /page/2 path exempts the call.

The crossing call — and the event — get code.file.path, code.line.number and code.function.name for the app line that made the call. instrument.http_client_origin decides which client spans pay for the backtrace: flagged (default, the crossing call), all (every call) or off. Under traces.details = tail, a trace with an N+1 of outgoing calls keeps its detail spans.

{ span.laravel.http.client.repeat = "n_plus_one" }
{ span.http.client.n_plus_one.count > 0 && kind = server }

The shape is taken from the request as the app built it; a withRequestMiddleware() callback that rewrites the URL afterwards is not seen by it.

What telemetry itself cost

The root span carries telemetry.overhead_ms: time spent in the package's per-operation hooks (queries, cache, model hydration) and the request middleware's own work, during that trace. Export happens after the response and is not included, nor is the framework's event dispatch around a listener — read it as a floor, not a full accounting.

Resource attribution

Request, worker-job and scheduled-task spans carry php.memory.peak_bytes and php.cpu.time_ms — the peak memory and CPU time of THAT unit of work (the process-global peak counter is reset per request/job/task, so long-lived workers report honestly). Matching histograms (http.server.memory.peak, http.server.cpu.time, queue.job.*, schedule.task.duration) give p95 memory/CPU per route, per job — and per custom label dimension. Disable with instrument.resources.

With cboxdk/system-metrics installed, spans additionally carry the process' real OS footprint via a ProcessMetrics tracker around each unit of work: process.memory.rss_peak_bytes (sees non-PHP allocations the PHP allocator misses) and process.cpu.utilization — the same mechanism cboxdk/laravel-queue-metrics uses for per-job metrics.

Every sub-span also carries its own php.cpu.time_ms and php.memory.delta_bytes (allocation delta — may be negative), so the trace waterfall shows WHERE the CPU and memory went, not just the totals. Backdated query spans are excluded (their work already happened when they're recorded).

{ name = "order.payment" } | select(span.php.cpu.time_ms, span.php.memory.delta_bytes)

{ kind = server && span.php.memory.peak_bytes > 134217728 }  # requests over 128 MB

Custom dimensions (context)

Faceted trace search — set the dimensions once, applied everywhere:

// e.g. in middleware, after tenant/team resolution:
Telemetry::context([
    'team.id' => $team->id,
    'team.name' => $team->slug,
    'plan' => $team->plan,
]);

A null value means not set: it removes the dimension rather than recording an empty one, so an optional value needs no filtering at the call site, and a dimension can be cleared without resetContext() (which would drop the trace continuation with it).

Telemetry::context(['tenant.id' => $tenant?->id]); // absent when null
Telemetry::context(['tenant.id' => null]);         // removes it again

From that point every span, event and telemetry-channel log record in the request carries the dimensions (span-specific attributes win on conflict) — and dispatched jobs inherit them, together with messaging.origin.name (the dispatching request/command name), so a job is queryable by team AND traceable back to the exact request that queued it:

{ span.team.name = "checkout" && kind = consumer }
{ span.messaging.origin.name = "POST /demo/orders" }

Context clears automatically between requests and jobs. It also crosses a real HTTP service boundary via the W3C baggage header — see Context propagation below.

Laravel Context (opt-in)

Telemetry::context() is the package's own, deliberately small dimension set. Apps often already keep request facts in Laravel's Context (Context::add('tenant', …)). Turn on instrument.laravel_context to read it onto root spans, telemetry-channel log records and exception records as laravel.context.* (keys snake-cased: tenantId → laravel.context.tenant_id; arrays redacted, then JSON-encoded).

• Hidden context (Context::addHidden()) is never read.
• instrument.laravel_context_keys limits it to the listed keys; at most 32 keys per record either way.
• It never becomes a metric label.

Laravel already carries Context into queued jobs, so a job's root span gets the same values.

Metric dimensions (bounded!)

Context is traces/events/logs only — metric labels multiply cardinality. For bounded dimensions (plan, tier, team — never raw ids) opt in to extra request-duration labels:

Telemetry::labelRequestsUsing(fn ($request) => [
    'plan' => $request->user()?->plan ?? 'guest',
]);

That enables per-plan latency in PromQL:

histogram_quantile(0.95, sum by (le, plan)
  (rate(http_server_request_duration_seconds_bucket[5m])))

Core labels (http.route, method, status) always win over resolver labels; a throwing resolver is reported and ignored.

Context propagation

Outbound propagation uses the full W3C traceparent — trace id and span id — so downstream spans are children, not detached roots:

• Queued jobs: payloads automatically carry the dispatcher's traceparent; workers continue it. (Sync jobs run inline in the dispatcher's context.) A job can opt out of the parent, never the link: see jobs that start their own trace.
• Incoming HTTP: the middleware continues traceparent headers when traces.continue_incoming is on.
• Outbound HTTP: opt in per request with the client macro (deliberate, so trace headers never leak to third parties by accident):

Http::withTraceparent()->post($url, $payload);

The macro is a no-op when no trace is active.

• Child processes: anything started through the Process facade gets TRACEPARENT in its environment — OpenTelemetry's carrier for process → process (traces.propagate_to_processes, default on; a value you set with Process::env() wins). An artisan command started with it set continues the trace as a child (traces.continue_from_environment, default on; needs instrument.commands to record anything), and so does any OTel-instrumented program. exec()/shell_exec() and Symfony's Process used directly are not covered. The facade's factory is only replaced when nothing else has bound it.

Telemetry::context() dimensions travel the same way — the macro also attaches a W3C baggage header (team.id=42,plan=pro, percent-encoded) whenever context is set, so a downstream SERVICE inherits the SAME custom dimensions, not just the trace id. The receiving app merges an incoming baggage header back into its own context (instrument.baggage, default on) — gated on traces.continue_incoming too, since baggage is caller-supplied, unvalidated data and should follow the same trust boundary as continuing the trace itself.

Behind an edge that injects traceparent

Some load balancers and proxies set a traceparent on every request so their access log carries a trace id that joins the backend's trace, but export no span themselves. Continuing such a header makes every request span the child of a span that never arrives, and trace backends list those traces as missing their root.

TELEMETRY_TRACES_CONTINUE_INCOMING_PARENT=false

The request then adopts the incoming trace id (the edge's log line still joins the trace) and the incoming sampled flag (subject to trust_incoming_sampling), but starts as the root span, carrying a span link to the span id the edge named. Leave it on (the default) whenever the caller does export that span, a service calling you with Http::withTraceparent() or an edge that traces itself, so your request span hangs under it.

Span links (retries)

Not every causal relationship is a parent. A retried job's attempt N+1 is a SIBLING of attempt N — both are children of the original dispatch span, not a continuation of one another — so nesting them as parent/child would misrepresent the shape. Instead (instrument.queue_retry_links, default on), attempt N+1's span carries an OTel span link back to attempt N's span:

{ span.queue.retry = true }

The link is bridged via the app's own cache (queue.retry_link_store/ queue.retry_link_ttl, default 86400s), keyed by the job's stable UUID — a retry can land on a different worker process, so this can't be in-memory state. A null/array cache driver just means retries go unlinked, same graceful degradation as everything else here.

Jobs that start their own trace

A job's span is a child of the span that dispatched it, and for most jobs that is the right shape. For two kinds it is not: a job delayed by hours stretches the request's trace across those hours, and a request that fans out thousands of jobs becomes one trace of thousands of spans. Mark those jobs and each starts a new trace instead, as its root, with a span link back to the dispatching span:

use Cbox\Telemetry\Contracts\StartsLinkedTrace;

final class RebuildSearchIndex implements ShouldQueue, StartsLinkedTrace
{
    // ...
}

Put the marker on the class you wrote: a queued listener, mailable or notification works the same way, though Laravel wraps it in its own job (CallQueuedListener, SendQueuedMailable, SendQueuedNotifications). For classes you cannot edit, list them instead; a parent class or interface matches everything under it:

// config/telemetry.php
'queue' => [
    'linked_traces' => [
        \Vendor\Package\Jobs\SyncCatalog::class,
    ],
],

What changes, and what does not:

• The link carries the dispatching span's trace id, span id and sampled flag (exported as the link's W3C trace flags), plus queue.dispatch = true. It is there for every attempt whose payload carried a trace, so the job is always one click from the request that queued it. messaging.origin.name still names that request.
• The new trace is sampled by the worker's own root decision (traces.sample_rate), not the dispatcher's. That is the point for a fan-out: each job is kept or dropped on its own.
• Each marked job in a batch gets its own trace, linked to the span that pushed the batch. In a chain, the next job is dispatched by the worker that ran the previous one, so a marked job links that job's span.
• A retried attempt keeps both links: the dispatch and its previous attempt.
• Dispatcher context dimensions (Telemetry::context()) still travel with the job.
• Sync jobs ignore the marker. They run inline, inside the dispatcher's unit of work, and stay its children.

{ span.queue.dispatch = true }

The trace id as a support reference

The trace id doubles as the reference that ties error trackers, support cases and logs back to the trace:

• X-Trace-Id response header on every traced request (traces.response_header, set null to disable).
• Server-Timing response header (opt-in, traces.server_timing): db;dur=12.3;desc="3 queries", cache;…, redis;…, telemetry;dur=0.4, app;dur=48.1 — shown by browser DevTools next to the network timing. Off by default: it tells the client how long your database took.
• traceresponse response header (W3C Trace Context Level 2): 00-<trace id>-<server span id>-<flags>, so a client — telemetry-browser after an Inertia visit — can parent what follows to the span that served it (traces.traceresponse, default on). The flag is the sampling decision in force when the response is sent. Sent only while traces.response_header is on — turning that off means no trace ids leave the app. Both headers are skipped on publicly cacheable responses.
• Laravel Context: trace_id is added at trace start — Sentry (≥ 4.x), Flare and every log channel pick it up automatically. An explicit Sentry scope tag is set too (traces.share_context).
• Error pages: Telemetry::traceId() is available while the error view renders — show it as “quote this reference id to support”.

The full flows (Sentry → trace, support case → trace, error page recipe) live in Error tracking & support flow.

Labelling a request for before/after

Send X-Telemetry-Label: before-eager-load and the request's root span carries laravel.trace.label = "before-eager-load" — as a context dimension, so every span of the request and the jobs it dispatches carry it too. Two runs of the same endpoint can be found and compared (the telemetry-ui MCP compare tool takes labels) without copying trace ids around. The value is cut to 64 characters of [A-Za-z0-9._:/-] and stays off metric labels. Rename or disable the header with traces.label_header.

{ span.laravel.trace.label = "before-eager-load" }

Sampling

traces.sample_rate (0–1) decides once per trace, at the root. Children inherit the decision; remote callers' decisions are respected via the sampled flag. Unsampled spans still exist as context — ids propagate — but are never buffered or exported.

Error spans escape sampling (traces.always_sample_errors, default on): a 10%-sampled app still exports every failing span. The escaped span's trace may be partial — healthy siblings were dropped under the head decision.

Per-route overrides via the Sample middleware — the re-decision covers the whole active trace, including the still-open request span:

use Cbox\Telemetry\Http\Middleware\Sample;

Route::get('/health', HealthController::class)->middleware(Sample::never());
Route::post('/checkout', ...)->middleware(Sample::always());
Route::get('/feed', ...)->middleware(Sample::rate(0.01));

From application code, where a route is the wrong place to decide — force-keep an admin's traces, say:

if ($request->user()?->isAdmin()) {
    Telemetry::sample(true);   // or a rate: Telemetry::sample(0.01)
}

It re-decides the whole active trace from that point, the open root span included, exactly like the Sample middleware.

Directives: record more of one slice, for a while

A sample rate is right for the fleet and useless for the one customer on the phone. A directive turns recording up for exactly one slice of traffic — a route, a path, a user, a tenant, a job — and turns itself off again:

php artisan telemetry:watch --user=42 --minutes=120 --profile --reason="ticket 881"
php artisan telemetry:watch --route="checkout.*" --minutes=15
php artisan telemetry:watch --context=tenant.id=7
php artisan telemetry:watch --job="App\\Jobs\\Billing\\*"
php artisan telemetry:watch --list          # --remove=<id>, --clear

use Cbox\Telemetry\Directives\Directive;

Telemetry::directives()->add(
    Directive::user(42)->profile()->for(minutes: 120)->because('ticket 881'),
);

A matching unit of work is kept whatever the sample rate said — a Sample::never() route or Telemetry::sample(false) after the match cannot drop it — keeps its detail spans under traces.details = tail, and every span from the match on carries telemetry.directive = <id>. With ->profile() it also gets a CPU profile: the native extension's when installed (its profile is kept because the trace now is); without it, excimer, started at the match — for requests only, so a job directive profiles only where the native extension runs.

Spans that already ended in an unsampled trace before the match are gone: path and job matches lose nothing, route and user matches lose what ran before routing or authentication, and a profile started late is partial.

Every directive expires (an hour by default, at most directives.max_minutes), and at most directives.max are active. They live in the application's cache (directives.store), so every host sees them; with none active, the per-request cost is one APCu read (an empty list is remembered for directives.recheck_seconds). Without APCu under PHP-FPM it is one cache read per request — TELEMETRY_DIRECTIVES=false if you do not use them. The read is the package's own: it is not recorded as one of the app's queries, cache operations or Redis commands.

A new directive reaches a host's workers within recheck_seconds. Only a directive added from a process that shares their APCu segment — an FPM request, not artisan (apc.enable_cli is off by default) — clears the remembered empty list at once.

From another app — a telemetry UI or insights install that is not this application — set TELEMETRY_DIRECTIVES_SECRET, and GET|POST /telemetry/directives and DELETE /telemetry/directives/{id} appear. Every call is signed:

X-Telemetry-Timestamp: <unix seconds>
X-Telemetry-Signature: sha256=<hex HMAC-SHA256 of "<timestamp>.<METHOD>.<id>.<raw body>">

<id> is the directive id for DELETE and empty otherwise, so a signed list call cannot be replayed as a delete. Each signature is accepted once — re-sign on retry. Five minutes of clock skew is allowed. PHP clients can call DirectivesController::signature($secret, $timestamp, $method, $id, $body). A POST body is the directive: {"match":"route","value":"checkout.*","minutes":15,"profile":true,"reason":"…","created_by":"insights"} (key for context); the id is always assigned by this app.

Ignoring request paths

Sampling still measures a request — Sample::never() drops its spans, but the http.server.* metrics, the analytics page view and any failing span (via the error escape) remain. For traffic you don't want in your telemetry at all — an observability dashboard reading its own backend, a load-balancer probe — list the paths instead:

// config/telemetry.php — or TELEMETRY_HTTP_IGNORE_PATHS="health,telemetry-ui,telemetry-ui/*"
'instrument' => [
    'http_ignore_paths' => ['health', 'telemetry-ui', 'telemetry-ui/*'],
],

A package that mounts its own routes registers them itself, from a service provider's boot() — data only, nothing is resolved or matched until a request arrives:

if (class_exists(\Cbox\Telemetry\Facades\Telemetry::class)) {
    \Cbox\Telemetry\Facades\Telemetry::ignorePaths(['telemetry-ui', 'telemetry-ui/*']);
}

Both lists apply (Telemetry::ignoredPaths() returns the merged set). Patterns are Str::is() globs matched against the request path without its leading slash: health is exact, telemetry-ui/* covers everything below the prefix but not the prefix itself, horizon* covers both, and / is the site root. The check runs once, first thing in the request middleware.

This package ignores its own routes

The same advice applies to the package giving it, so it takes it: the Prometheus scrape endpoints, the browser span ingest, the RUM asset and the source map upload are ignored out of the box. A 15-second scrape is ~5,700 requests a day that measure nothing about your app, and it would sit near the top of your own route tables; the ingest route fires once per real page view, so telemetry would report itself as traffic.

Nothing is lost by it. Prometheus already records scrape_duration_seconds per target, from the side that can act on it.

The exclusion follows whatever path each route is configured with, so moving an endpoint moves it too. To measure them like any other route:

'instrument' => [
    'http_ignore_own_routes' => false, // TELEMETRY_HTTP_IGNORE_OWN_ROUTES=false
],

Your own routes stay yours. Laravel's /up is the usual next candidate — an uptime check hits it every few seconds — but it is not excluded by default: silently dropping traffic an installation already records is worse than the noise. Add 'up' to http_ignore_paths when you want it gone.

What an ignored request gets:

• No server span, no http.server.request.duration / .memory.peak / .cpu.time, no analytics.page_view, no X-Trace-Id response header, and no incoming traceparent/baggage is continued.
• No trace for anything inside it. The tracer is suppressed for the request rather than left without a root: the HTTP client, mail and notification instrumentations open spans with or without a parent, and each would otherwise become a trace root of its own — a dashboard's Tempo/Loki calls turning up as hundreds of orphan traces. Spans still open as context, so nothing starts a root; none is exported, not even an error span (the always_sample_errors escape would publish a child whose parent was never recorded). Telemetry::currentSpan(), traceId() and traceparent() return null, so nothing propagates: a job dispatched or a service called from the request starts its own trace.
• Exceptions are still reported. report() — and so an unhandled exception — still writes the exception record and increments exceptions.reported: errors in an ignored path must not vanish. The record carries no trace or span id, because there is no trace to link to.
• Fleet metrics from other instrumentation still count when they are independent of the request span — an outgoing HTTP call still records its client-duration histogram, a cache hit still counts. Those are real load on the backend, whoever caused it. Instrumentation that only records inside a request span (query spans and db.queries, Redis, views) records nothing.

The suppression is lifted when the request terminates (and by the Octane reset), so the next request on a long-lived worker is traced normally. Browser RUM (@telemetryBrowser) reports from the page itself — leave the snippet off pages you ignore.

Tail detail retention

MANY details when it hurts, a lean skeleton when all is well:

TELEMETRY_TRACES_DETAILS=tail
TELEMETRY_TRACES_SLOW_REQUEST_MS=1000
TELEMETRY_TRACES_SLOW_SPAN_MS=100

In tail mode, detail spans (cache operations, queries) are kept only for traces that turned out interesting: an error span anywhere, a request over slow_request_ms, or a single detail span over slow_span_ms (one slow query keeps the WHOLE trace's details). Healthy fast traces ship the skeleton — root span with all its tallies (db.query.count, cache.event.count, resources) — while counters and histograms flow unconditionally.

Only detail spans are trimmed: queries, cache and Redis operations, views, request phases, Livewire hooks, storage and broadcast calls. Every other span stays in every trace: the request, jobs, outgoing HTTP, mail, and the laravel/ai agent, model and tool spans.

Your own keep rules

Add the traces and spans that matter to you. A rule is a map of attribute to pattern, all of which must match (name is the span name; * and ? work, a list of patterns matches any one, case is ignored). Any matching rule wins.

// config/telemetry.php → traces.details
'keep_when' => [
    // A matching span keeps its whole trace's detail.
    ['http.route' => ['checkout*', 'api/orders*']],
    ['gen_ai.operation.name' => 'invoke_agent'],   // every agent run, with its queries
    ['tenant.id' => '7'],                          // one tenant, from Telemetry::context()
],
'keep_spans' => [
    // A matching detail span is kept in every trace, healthy or not.
    ['db.query.text' => ['insert *', 'update *', 'delete *']],
],

An attribute a span does not carry never matches, not even *. For a temporary, time-boxed version of keep_when — one user, one route, for an hour — use a directive instead.

The decision happens at flush, when the entire trace is in memory — tail-based detail retention without a collector. Buffer-cap force flushes always keep details: a 5000-span request IS interesting.

Bootstrap visibility

When LARAVEL_START is defined (it is, in every standard public/ index.php), the request trace includes a backdated laravel.bootstrap span covering framework boot until the request middleware runs, and the request span carries laravel.bootstrap_ms. The request middleware is appended to the global stack, so the span also covers the app's global middleware that runs before it.

Only the first request a process serves gets it, because that is the only request that waited for the boot. Under FPM every request is the first. A long-lived runtime that defines LARAVEL_START once per worker (NativePHP's persistent interpreter, a custom worker) records it once, for its first request. Octane doesn't define LARAVEL_START, so its requests have no bootstrap span.

Request phases

The request span splits into phases, taken from framework events, so the waterfall shows where the time went without a profiler:

Each one is a detail span (tail mode trims it from healthy fast traces) plus a tally on the request span that is always kept: laravel.routing_ms, laravel.middleware_ms, laravel.handler_ms, laravel.send_ms and laravel.terminate_ms. A boundary that never fires folds its phase into the next: a 404 matches no route, so it has no laravel.routing and its laravel.handler starts at the middleware.

Laravel announces no event between the middleware stack ending and the controller starting, so that boundary comes from decorating the controller dispatcher Route::run() resolves from the container. Two kinds of request never reach it and so have no laravel.middleware: a closure route, which is not dispatched through a controller, and a request a middleware answered itself (an auth redirect, a rate limit). Their time folds into laravel.handler, which is more honest than reporting a middleware cost of zero for a request that never got past the middleware.

Which code answered

The request span carries the action alongside the route, because http.route says which URL pattern matched, not which code ran — routes share controllers, and one controller answers many routes.

An invokable controller reports __invoke. A closure route has no class, so it gets code.function set to Closure and no code.namespace — inventing one would give a UI something false to group by. Both are bounded by the route table.

Laravel runs terminate() after the response has gone out. The request span covers that work, so spans from defer() callbacks stay in the request's trace. http.server.request.duration doesn't: it stops when the response is sent, because nobody waits for work after that.

Connect time

QueryExecuted and CommandExecuted fire only once a connection is already up, so the handshake — DNS, TCP, TLS, auth — is invisible to query instrumentation. It is also where a healthy-looking app spends its worst seconds: a database that answers every query in a millisecond still hangs for thirty if the connect blocks, and in a query-only waterfall that shows up as an unexplained gap before the first db.query.

db.connect and redis.connect fill that gap. Laravel resolves PDO lazily, so the span is emitted the first time a connection is actually used and not again — a request making two hundred queries has one db.connect, not two hundred. The same lazy resolution is why the timing wraps the resolver closure rather than ConnectionFactory::make(), which returns before any socket is opened.

Redis depends on the client:

• phpredis opens the socket in Laravel's connector, so the connector is timed.
• predis opens it on the first command, long after the connector has returned. Timing the connector would report a few microseconds of object construction, so instead the client is handed a connection factory whose connections time their own handshake (DNS, TCP, TLS and the AUTH/SELECT init commands) when they actually connect. Every node predis talks to goes through that factory, so under Sentinel a cold request shows two redis.connect spans: one for the sentinel it asked, one for the master it was sent to. server.address is the node actually dialled. A connection that sets its own predis connections option keeps its own factory and is not timed, and neither is a scheme mapped to anything other than predis's stream connection (Relay, a custom class).

These spans are not detail-marked. A connect is rare and high-signal, so it survives traces.details.mode=tail trimming — which is exactly when you want it, since a trace is trimmed for being healthy and kept for being slow.

A connect that throws is recorded and then rethrown unchanged: a twenty-second connect that ends in a refused connection is the most useful span in the trace, and losing it to the exception would be losing the answer.

Half-open client spans

A client span is opened on one framework event and closed on another — RequestSending / ResponseReceived, MessageSending / MessageSent. Where the framework hands back a different object than it started with, the two cannot always be paired, and an unpaired span is neither ended nor exported.

The pairing is by identity wherever an identity survives: outgoing HTTP keys on the PSR-7 request, which both Request wrappers share, so it is exact even for concurrent Http::pool() calls to the same URL. Mail has no such key — Laravel sets no Message-ID before dispatching MessageSending, and most transports clone the message — so it falls back to the innermost open send.

Deliberately, nothing guesses by what the call looks like. Matching a failure to a span by method/host/path closes whichever lookalike happens to be open, and two concurrent calls then swap their statuses and durations. A missing span is much the lesser evil than one carrying someone else's outcome, which reads as data.

These paths therefore go unmatched today:

What it costs, and where

The consequence is twofold: that one call is missing from the trace, and while the span sits on the context stack, later work in the same unit of work is parented under it.

ManagesRequestState::flushRequestState() drops the half-open state, and it runs on Octane/NativePHP request and tick boundaries, and at the start of each non-sync queue job (the latter only when instrument.queue is on). Those are the boundaries that bound it.

Under FPM there is nothing to bound — the process ends after the request, so nothing survives to accumulate.

The case with no boundary is a long-running CLI process that is neither Octane nor a queue worker: an artisan command looping over thousands of redirected HTTP calls, or mail sends whose transport keeps throwing, holds one entry per unmatched operation for the life of the command. If you have such a command, give it explicit boundaries — process in chunks and let the work happen in queued jobs rather than inline, which restores the per-job reset.

Buffering

Finished spans buffer in memory and flush at terminate — export latency happens after the response is sent. The buffer is capped (traces.max_buffer, default 5000) and force-flushes when full, so long-running workers and Octane can't grow unbounded.