Job Lifecycle and Attempts

Queue Monitor records one row per attempt. A job that runs once has one row; a job that is retried, or redelivered by the broker, has one row per attempt. All rows of one job share a job_id and are linked through retried_from_id, and together they form the job's retry chain (QueueMonitor::getRetryChain($uuid)).

This page explains how rows are created and closed, what each status means, and how Queue Monitor keeps every outcome on the row of the attempt that produced it, even when two attempts of the same job run at the same time.

Identifying a Job

job_class and display_name

job_class is always the class name of the command that was queued. display_name is a label, and is often the same string. Group, filter and alert on job_class, and show display_name to people.

Closures, mailables, notifications and listeners are queued inside a framework wrapper, and that wrapper is the class the worker runs. To find every run of one mailable or listener, filter on display_name.

Two cases need care:

• Sync jobs. These are dispatch_sync() calls and jobs on the sync connection. Laravel runs them inline without raising JobQueued, so their row is created when they start processing, from the queue payload. The class is taken from the payload's data.commandName. Earlier versions took the payload's display name as job_class for these rows, so a job with a custom displayName() was recorded under a class name that does not exist. Rows recorded before the upgrade keep that value.
• Jobs without a displayName() method. For a row recorded when the job was queued, display_name is null. For a row first recorded when the job started processing (sync jobs, for example), display_name is the class name, because Laravel writes that into the payload.

Statuses

How Each Queue Event Is Recorded

Released Jobs

A job released back onto the queue fires JobProcessed, but it has not finished. A release comes from rate limiting, job middleware or $this->release(). Its row is parked as queued, and the next delivery reuses the same row and updates its attempt number. A release therefore does not leave a duplicate row or a false failure.

The database driver releases a job by deleting its row from the jobs table and inserting a new one, so the next delivery has a new job id. Queue Monitor connects the two through the payload uuid, which survives the release, in Laravel's cache:

• After a plain release, the next delivery takes over the parked row: the row moves to the new job id and is reused, as on every other driver.
• After a release on an exception, the failed attempt keeps its row and its job id, and the next attempt's row links to it through retried_from_id.
• The new jobs row exists as soon as the job is released, so another worker can start the next delivery before the release is recorded. Queue Monitor connects the two whichever is recorded first.
• Only the released attempt's own row moves. A slow attempt that the broker redelivered under the same id keeps its row, and its late outcome lands there. Per-job CPU and memory reported under the old id still find a row that moved.

This needs a cache store that every worker shares, which is what Laravel's own queue:restart relies on too. The store is queue-monitor.cache.store, or the default store when that is not set. With the array store, or a file store across several hosts, a released row stays queued. A job released with a delay longer than seven days is not connected to its next delivery.

Every other driver keeps the job id across a release.

Intermediate Failures

Laravel only fires JobFailed for the final attempt. When an earlier attempt throws, the worker raises JobExceptionOccurred and then releases the job for a retry. Queue Monitor closes the attempt's row as failed on JobExceptionOccurred, with the exception and the duration of the attempt itself. The backoff and the wait until the next delivery are not part of it. The next attempt gets a row of its own, linked through retried_from_id.

For the final attempt the worker raises JobFailed first, which records the failure, and JobExceptionOccurred after it. The second event leaves the row alone, so the exception and duration are recorded once.

The sync driver, and the deferred and background drivers built on it, raise the two events the other way round: JobExceptionOccurred first, then JobFailed. The row is closed as failed on the first, and the second keeps the duration already recorded. A final attempt that times out raises JobFailed and then JobTimedOut; the row ends as timeout, also keeping the duration recorded on JobFailed.

Rows recorded by versions before 1.12.1 left an intermediate failure processing until the next attempt started. Such a row is still closed as failed when the next attempt starts, with no duration, because the time until then includes the backoff.

Retried Jobs

queue:retry pushes a failed job back onto its queue. On SQS and the database driver the job gets a new id, so the retry is recorded as a new job with a row of its own.

On Redis the retried job keeps its id, with its attempts reset, so it starts again at attempt 1 under a key that already has a finished retry chain. The finished rows are left as they are: the retry run gets new rows, starting at attempt 1, and the first is linked through retried_from_id to the row recorded last. The job's retry chain then holds both runs in the order they ran, and the same attempt number can appear once per run.

Earlier versions rewrote the final failed row as the retry's attempt 1, which erased the failure.

Inferred Timeouts

Some attempts never report back. A worker can be killed by SIGKILL, the OOM killer, or a supervisor or container timeout. Its attempt then raises no JobProcessed, JobFailed or JobTimedOut event, and its row stays processing. Once the broker's redelivery window passes, the message is delivered again:

When the redelivered attempt starts and finds the previous attempt still processing with no exception, it closes that attempt as timeout with no duration and opens a row of its own. The time until redelivery includes the broker's wait, so recording it as a duration would inflate duration statistics.

This timeout is an inference. It means "a newer attempt started while this one had not reported back". That is not proof that the worker died.

Slow Jobs That Outlive the Redelivery Window

The same signal appears when a job is merely slow. If it runs longer than the visibility timeout or retry_after, the broker hands it to a second worker while the first is still running it. Both attempts are then in flight at once:

t=0s     worker A starts attempt 1           row 1: processing
t=90s    retry_after lapses; worker B starts attempt 2
                                              row 1: timeout (inferred)
                                              row 2: processing
t=150s   worker A finishes attempt 1         row 1: completed, duration 150 s
t=170s   worker B finishes attempt 2         row 2: completed, duration 80 s

When the slow attempt finally reports, its real outcome replaces the inferred timeout:

• JobProcessed marks it completed with its real duration, measured from its own start.
• JobFailed marks it failed with the exception and real duration.
• JobExceptionOccurred, for an attempt with retries left, marks it failed with the exception and real duration. Nothing else would close that row again.
• JobTimedOut marks it timeout with its real duration and a JobTimeout exception.
• Per-job CPU and memory from laravel-queue-metrics land on the slow attempt's row.

The newer attempt's row is not touched. It keeps its own outcome, whichever attempt finishes first.

An attempt whose worker really was killed never reports, so it stays timeout with no duration.

If a slow, superseded attempt releases itself, its row is left as it is rather than parked as queued. The next delivery continues on the newer attempt's row, so a parked row would stay queued forever.

A Late SQS Delete That Fails

When a slow SQS job finishes after its visibility timeout, the first worker deletes the message with the receipt handle it received. That handle belongs to a delivery SQS has since replaced, so whether the delete works depends on the service:

• Amazon SQS usually accepts the old handle and deletes the message. The slow attempt is recorded as completed, as in the timeline above.
• ElasticMQ and some other SQS emulators reject it with ReceiptHandleIsInvalid. The worker raises that error after handle() returned, so Laravel treats the attempt as having thrown, and Queue Monitor records it as failed with the ReceiptHandleIsInvalid exception. The job's own work did complete.

Either way the job ran twice, once per attempt, which is the problem to fix. A failed attempt whose exception is ReceiptHandleIsInvalid (or another error from DeleteMessage) is the signature of a visibility timeout that is shorter than the job.

Avoid redelivering running jobs. A slow job that is redelivered runs twice. Set the queue's retry_after (or the SQS visibility timeout) longer than the longest job can take, and keep the worker --timeout below it. See the Horizon guide for the same rule under Horizon.

Durations

duration_ms is measured by the worker that ran the attempt, from the moment it started the attempt to the moment its outcome is recorded, to the millisecond. The started_at column only holds whole seconds, so it is used only when the outcome is recorded by a process that did not see the start: a stuck-job resolution, or a worker that restarted in between. A duration measured that way can be up to a second too long.

When one attempt reports two outcomes — JobExceptionOccurred then JobFailed on the sync driver, JobFailed then JobTimedOut for a final attempt that times out — the duration is measured once, on the first, and the second keeps it.

How Attempts Are Matched

Each outcome is written to the row matching both the job key and the attempt number the worker is running. The worker's attempt number is Job::attempts():

If no row carries that attempt number, the outcome goes to the job's latest attempt, which is how earlier versions matched every outcome. This happens when the row was pruned, the broker over-reported a count, or a custom repository cannot look attempts up.

Two sources do not carry an attempt number of their own:

• Per-job metrics. laravel-queue-metrics raises JobMetricsCompleted and JobMetricsFailed with the job key only. Both are raised inside the worker that ran the job. When a job starts, Queue Monitor remembers in that worker process which attempt it began, and uses that attempt when the metrics arrive. If the process never saw the start, it falls back to the attempt that is still processing.
• Beanstalkd after deletion. Beanstalkd's attempts() asks the server, which fails once a finished job has been deleted. The attempt this worker remembered at the start is used instead.

Stuck Jobs

A stuck job is a row that has been processing for longer than health.stuck_job_minutes. Resolving one marks it timeout, and with the retry action also replays it. See Stuck Jobs.

Redelivery and stuck-job resolution do not get in each other's way:

• An attempt closed as timeout by a redelivery is no longer processing. The health check does not count it as stuck, and the resolver leaves it alone.
• A stuck row you resolve is also an inferred outcome. If the job turns out to have been slow and finishes later, its real outcome replaces the timeout in the same way.

Custom Repositories

Attempt lookup lives in its own contract, Cbox\LaravelQueueMonitor\Repositories\Contracts\JobAttemptRepositoryContract, beside JobMonitorRepositoryContract. Adding a method to JobMonitorRepositoryContract would break every host that implements it. The default EloquentJobMonitorRepository implements both contracts.

A custom repository that implements only JobMonitorRepositoryContract keeps working, and outcomes go to the latest attempt as before. That is right whenever one attempt of a job runs at a time. When a slow job is redelivered while its first worker still runs it, two things land on the wrong row:

• the slow attempt's status, duration and exception go to the newer attempt's row;
• its CPU and memory from laravel-queue-metrics go to the attempt still processing, which is the newer one, so the newer attempt shows the slow attempt's figures.

To get per-attempt matching, implement the second contract as well:

use Cbox\LaravelQueueMonitor\Models\JobMonitor;
use Cbox\LaravelQueueMonitor\Repositories\Contracts\JobAttemptRepositoryContract;
use Cbox\LaravelQueueMonitor\Repositories\Contracts\JobMonitorRepositoryContract;

class CustomJobMonitorRepository implements JobAttemptRepositoryContract, JobMonitorRepositoryContract
{
    public function findAttemptByJobId(int|string $jobId, int $attempt): ?JobMonitor
    {
        return JobMonitor::where('job_id', (string) $jobId)
            ->where('attempt', $attempt)
            ->orderByDesc('id')
            ->first();
    }

    // ... the JobMonitorRepositoryContract methods
}