Queue workers
Queue workers
Run exactly one php artisan queue:autoscale per host, always. Without it, nothing
queued is ever sent: no webhook, no back-channel logout to the apps somebody signed out
of, no synced app roles, no Postal delivery report, no queued email. The web tier keeps
answering 200 the whole time, so nothing else tells you.
What runs on the queue
Everything below is dispatched to a queue and done by a worker, never while the person waits:
| Work | Job | Queue setting |
|---|---|---|
| Webhook deliveries | DeliverWebhook |
CBOX_ID_WEBHOOKS_QUEUE_CONNECTION / CBOX_ID_WEBHOOKS_QUEUE |
| Back-channel logout tokens | DeliverBackchannelLogout |
the default queue |
| App role manifests pulled from apps | SyncAppManifestJob |
the default queue |
| Outbound user sync and audit streaming | DrainProvisioningConnection, PumpAuditStream, PumpStreamDeliveries |
the default queue (SIEM_QUEUE* for the last) |
| Push notifications to trusted devices | DeliverPushNotification |
CBOX_ID_DEVICES_QUEUE_CONNECTION / CBOX_ID_DEVICES_QUEUE |
| Postal webhooks and inbound mail | ProcessWebhook, ProcessInboundMessage |
POSTAL_WEBHOOK_*, POSTAL_INBOUND_* |
| Queued mail, notifications and listeners | the framework's own | the default queue |
A blank setting means "the default queue of the default connection" (QUEUE_CONNECTION
and its REDIS_QUEUE/DB_QUEUE, normally default). Out of the box every job lands on
one queue: redis:default in production.
You do not list these anywhere. config/queue-autoscale.php derives the set from the
same settings the dispatchers read (App\Platform\Queues\DispatchedQueues), and gives
each connection one worker group that polls all of its queues, default queue first. Move
webhooks to CBOX_ID_WEBHOOKS_QUEUE=hooks and the workers follow on the next manager
start.
The manager
cboxdk/laravel-queue-autoscale is the worker supervisor. It is one long-running process
that starts queue:work children, sizes them against a pickup SLA, and stops them again.
You never run queue:work yourself, and you do not configure a separate queue-worker
process next to it: two supervisors fight over the same jobs.
The settings are in config/queue-autoscale.php and App\Platform\Queues\WorkerProfile,
with the reason for each value beside it. The defaults are sized for the smallest shape
this runs on, a 512 MB host that also serves the web traffic; the production cluster
changes the first row (below):
| Setting | Value | Why |
|---|---|---|
| Mode | single host (QUEUE_AUTOSCALE_CLUSTER_ENABLED=false); cluster in production |
One host, one manager. Cluster mode elects a leader over Redis/Valkey, for when more than one manager can run. |
| Workers per group | min 1, max 2 (QUEUE_AUTOSCALE_WORKERS_MIN / QUEUE_AUTOSCALE_WORKERS_MAX) |
Never scale to zero: a cold start on every sign-out is latency for nothing. Raise the max with the total cap. |
limits.max_total_workers |
2 (QUEUE_AUTOSCALE_MAX_TOTAL_WORKERS) |
The hard cap that keeps PHP-FPM alive. A worker of this app is about 75 MB resident, the manager about the same. |
limits.max_memory_percent |
70 | Stop spawning while the instance is above 70 % memory. |
| Pickup SLA | 30 s | Also the line the queue_workers health check turns red at. |
| Job timeout | 75 s | Must stay below retry_after (90 s), or Redis hands a running job to a second worker and a webhook or logout token is sent twice. |
| Failure fuse | on | A relying party that is down gets fewer workers, not more. |
Check the configuration against the queues that exist:
php artisan queue:autoscale:doctor
A healthy production run prints Discovered queues: 1 and ✓ Nothing to report. Before
anything has been dispatched it says there is nothing to check yet, which is expected.
queue:autoscale needs ext-pcntl and ext-posix. Composer refuses to install without
them; php artisan cbox-id:doctor checks them again in the CLI the manager runs under and
says so in words if either is missing.
Kubernetes
Production (cboxid.com) runs the manager in a worker Deployment of its own: one pod,
the web pods' image and environment, with nginx and PHP-FPM switched off and cbox-init
running two processes in it — php artisan queue:autoscale and the scheduler
(php artisan schedule:work, which also runs the monitor's retention, below):
env:
- {name: CBOX_INIT_PROCESS_PHP_FPM_ENABLED, value: "false"}
- {name: CBOX_INIT_PROCESS_NGINX_ENABLED, value: "false"}
- {name: CBOX_INIT_PROCESS_SCHEDULER_ENABLED, value: "true"}
- {name: CBOX_INIT_PROCESS_AUTOSCALER_ENABLED, value: "true"}
- {name: CBOX_INIT_PROCESS_AUTOSCALER_USER, value: www-data}
- {name: CBOX_INIT_PROCESS_SCHEDULER_USER, value: www-data}
- {name: CBOX_INIT_PROCESS_AUTOSCALER_SHUTDOWN_TIMEOUT, value: "80"}
- {name: CBOX_INIT_GLOBAL_SHUTDOWN_TIMEOUT, value: "80"}
cbox-init restarts either process if it exits. The workers the manager starts live
inside that pod, so the web pods' memory is not what they compete for.
Deploys: nothing to add. A rollout replaces the pod — Recreate, not overlapped, so
there is one scheduler at a time: Kubernetes sends the outgoing pod SIGTERM, the manager
lets its workers finish their current job within the 80-second shutdown timeout (inside a
90-second termination grace), and the new pod starts a manager on the new code.
php artisan queue:autoscale:restart would only restart the outgoing manager on the
outgoing release.
What the environment sets, and why:
QUEUE_CONNECTION=redisandCACHE_STORE=redis, both on the namespace's Valkey. The manager and the web tier meet in the cache: the heartbeat, the failure fuse and the restart signal all live there.QUEUE_AUTOSCALE_CLUSTER_ENABLED=true. One manager runs, but in cluster mode a second — a replica added later, or one started by hand in a pod — elects a leader over Valkey with it instead of each sizing workers as if alone.- The Valkey runs
maxmemory-policy noeviction. A queued job is data: under an evicting policy a full instance silently deletes webhooks and mail. - Queue metrics use Redis by default (
QUEUE_METRICS_STORAGE=redis); leave it.
On another cluster, run the same thing: either this one worker pod, or the manager and the
scheduler as two Deployments of one replica each (the local cbox.yaml does the latter,
as queue and scheduler processes).
Sizing. Two workers is a ceiling chosen for a 512 MB host shared with the web tier, and the worker pod shares with nobody. There are two ceilings, and you raise them together:
QUEUE_AUTOSCALE_MAX_TOTAL_WORKERS, the hard cap for the whole host (pod).QUEUE_AUTOSCALE_WORKERS_MAX, the most workers one group runs. Every queue this app dispatches to on one connection shares one group, so this is the number that actually decides how many workers run. Raising the total cap alone changes nothing.
QUEUE_AUTOSCALE_WORKERS_MIN sets the floor (default 1, never below 1). For example, a
worker pod with a 1 GiB limit, in the environment it reads:
env:
QUEUE_AUTOSCALE_MAX_TOTAL_WORKERS: "6"
QUEUE_AUTOSCALE_WORKERS_MAX: "6"
The app checks the pair when its configuration loads and refuses to start, naming the
variable, if WORKERS_MAX is below WORKERS_MIN or above the total cap (a group maximum
the cap would silently clamp). The web pods read the same configuration, so a bad pair stops
the rollout at its readiness probe instead of shipping a setting that does nothing.
Budget about 96 MB per worker plus the manager's own 75 MB against the pod's memory limit.
Keep a hard cap even then: it holds when a container reports the node's memory rather than
its own limit, which makes max_memory_percent meaningless.
On a PaaS
A platform that runs long-lived processes beside the web ("background process", "worker
process", a Procfile worker: line) runs the manager the same way: one process,
php artisan queue:autoscale, restarted if it exits. Two things to check:
- Do not also add the platform's own queue-worker type. The manager is the worker; two supervisors fight over the same jobs.
- Find out whether it runs one per instance. Many platforms start a background process
on every instance of the app. If the app scales past one, either move the manager to a
worker group with exactly one instance, or set
QUEUE_AUTOSCALE_CLUSTER_ENABLED=trueso the managers elect a leader and share the work.
If the platform replaces instances on deploy and sends SIGTERM first, there is no deploy
step to add; if it restarts processes in place, keep php artisan queue:restart in the
deploy, as below.
On a VM (systemd or Supervisor)
Run the manager under your process supervisor, one per host. systemd:
[Unit]
Description=Cbox ID queue manager
After=network.target
[Service]
User=www-data
WorkingDirectory=/var/www/cbox-id/current
ExecStart=/usr/bin/php artisan queue:autoscale
Restart=always
RestartSec=5s
KillSignal=SIGTERM
TimeoutStopSec=60s
[Install]
WantedBy=multi-user.target
TimeoutStopSec must be at least the workers' 30 s drain window, so the manager can stop
its workers before systemd kills it. Supervisor works the same way (stopsignal=TERM,
stopwaitsecs=60).
On deploy, after the new release is in place:
php artisan queue:restart
The manager honours Laravel's own restart signal: its workers finish their current job and
exit, the manager exits, and the supervisor starts it again on the new code.
php artisan queue:autoscale:restart restarts only the manager.
docker-compose.yml already runs the manager as the queue service.
The manager needs Redis for its metrics (QUEUE_METRICS_STORAGE=redis, the default). A
single-host install without Redis can switch queue metrics to database storage, but that
is not set up in this repository.
The health signal
GET /health/status?token=… (the HEALTH_TOKEN, as for readiness) runs a queue_workers
check, and answers 503 when it is red. It is red when either is true:
- No manager has reported in for
2 × scaling.cooldown_seconds + 4 × interval(140 s by default). The manager beats on every evaluation cycle; the window is long enough that a manager holding a scale-down, which says nothing for up to one cooldown, never reads as dead. - A supervised queue's oldest waiting job is older than its pickup SLA (30 s). This
is read from the queue itself, so it catches workers that are running but not getting
through, and it still works if you run plain
queue:workwith the autoscaler off (QUEUE_AUTOSCALE_ENABLED=false; the manager half then stands down).
The response carries counts, ages and queue names only, never job contents.
Route on /up and /health/ready; alert on /health/status. Readiness answers one
question: can this instance serve web traffic? The queue manager is a separate process (on
Kubernetes a separate pod), and if its death made readiness red, every web instance would
be taken out of the load balancer at once. So queue_workers is deliberately not on
/up or /health/ready; it is on /health/status, which nothing routes on. Point uptime
monitoring and alerting there.
One thing to know when a red clears by itself: on Redis, a job released with a delay (a back-channel logout retry) keeps its original creation time, so for the few seconds between becoming due and a worker taking it, it can read as older than the SLA. A red that lasts is real.
php artisan cbox-id:doctor reports the same state in words.
The job monitor
Platform › Queues (/admin/platform/queues) shows whether a manager is running
and how far behind each queue is. Open job monitor on that page opens the
cboxdk/laravel-queue-monitor dashboard at /admin/platform/queues/monitor: every job with its
status, duration, retries and failure message, and the autoscaler's scaling decisions.
Who can open it:
- Platform operators only. Anyone else signed in gets a 404; a signed-out visitor is sent to sign in.
- On the platform root's host only. On a customer environment's host, including a white-label domain, the address does not exist, for operators too.
- The package's own address (
/queue-monitor) is not registered.
The dashboard runs Alpine.js, which needs 'unsafe-eval' and an inline script, so these
routes carry a Content-Security-Policy of their own. Only script-src is wider than the
console's; everything else stays same-origin.
Job payloads are never stored. A queued job is its constructor arguments: webhook
bodies, and in whatever job is added next, tokens or email addresses. The package would
keep them for replay, and its redaction only covers what its screens show. Here payload
storage is off in config/queue-monitor.php (as a literal, not an environment variable),
and the column is blanked on every write in App\Providers\QueueServiceProvider, even for
a job that asks for its payload to be kept. The price is replay from the dashboard: retry
failed jobs with php artisan queue:retry, which reads Laravel's own failed_jobs table.
The monitor still stores failure messages and stack traces. Keep job exceptions free of
secrets, the same rule failed_jobs already needs.
The REST API is off. If you turn it on (QUEUE_MONITOR_API_ENABLED=true), it sits
under /admin/platform/queues/api behind the same operator session and host check as the
dashboard.
Retention runs on the scheduler:
| Schedule | Command | Keeps |
|---|---|---|
| daily | queue-monitor:prune |
7 days of job history, at most 100,000 rows (QUEUE_MONITOR_MAX_ROWS), and the autoscaler's scaling events |
| every 15 min | queue-monitor:resolve-stuck |
marks a job still processing after 15 minutes as timed out; a worker died mid-job |
The tables are created by the package's own migrations (queue_monitor_jobs,
queue_monitor_tags, queue_monitor_scaling_events, queue_monitor_cluster_events) and
run on SQLite, PostgreSQL and MySQL.
Troubleshooting
/health/status says no manager has ever reported in. The manager is not running, or it
runs against a different cache than the web tier (check CACHE_STORE on both). On
Kubernetes, check the worker pod is running and read its log; on a PaaS, the background
process.
The manager runs, but jobs still wait. php artisan queue:autoscale:debug --queue=default --connection=redis shows the metrics it sees and the fuse state. An open
fuse means most recent jobs failed; the monitor shows why.
Failed to spawn worker in the log. Run php artisan queue:work redis --queue=default
by hand as the same user; the error is usually obvious.
Two managers on one host. The second exits with a lock error. Use
php artisan queue:autoscale --replace to take over from a stuck one.