Prometheus Metrics
Prometheus Metrics
Comprehensive Prometheus metrics for monitoring Cbox Init and managed processes.
Configuration
Enable metrics in your cbox-init.yaml:
global:
metrics_enabled: true
metrics_port: 9090
metrics_path: /metrics
Non-waking FPM observations (unreleased)
PHP baseimages already provide FPM and OPcache metrics through fpm-exporter.
Reuse that exporter; cbox-init does not implement a second FPM collector.
For snapshot workloads, the optional CBOX_SNAPSHOT_OBSERVE_LISTEN address
provides a monitoring connection to CBOX_SNAPSHOT_UPSTREAM without requesting
restore or extending the customer idle timer. For example, with the ordinary
service proxy configured on 127.0.0.1:9001, use 127.0.0.1:9002 for observations
and point the exporter's FastCGI connection at that observation port.
Only explicit loopback IP addresses are accepted. This is privileged raw upstream access, not HTTP, and must never be exposed through ingress or a Service. Processes sharing the pod network can reach it. An admitted probe pins the workload against checkpoint while its connection is open, for at most five seconds. Probes arriving during sleep/checkpoint are disconnected without wake. Missing samples must be represented as unavailable, not as zero or disabled OPcache. All monitoring traffic must use this path; direct upstream connections bypass its coordination.
Run make test-e2e-fpm-observation to verify real FPM status and enabled OPcache
through the observation proxy using the Cbox PHP baseimage. The test checks that
probes leave the customer idle clock untouched, are refused while asleep without
wake, and resume after the awake transition. It prints the tested image ID; set
CBOX_TEST_PHP_IMAGE to select another image or immutable digest. Its lifecycle
control is fake: it does not perform a CRIU checkpoint or restore.
Verified locally on 2026-11-04 with race detection:
- PHP 8.4 Bookworm, image ID
sha256:9e0d7787d5646436dc3839090ba6fd18a25da973e9e1db4cfa06f513d56900ba. - PHP 8.5 Bookworm rootless, image ID
sha256:b4d7228dc08320fa8464f961725ae06f0216dcf8f547e8c7e0d9430dede63c20.
These are local Docker image IDs, not registry manifest digests. By default the test runs this checkout's proxy with the existing phpfpm dependency. To include the actual exporter process, build the selected fpm-exporter checkout separately and pass its absolute executable path:
CBOX_TEST_FPM_EXPORTER_BINARY=/absolute/path/to/fpm-exporter make test-e2e-fpm-observation
The runner prints the binary SHA-256. This requires the unreleased exporter observation-quality changes (9e2b980 or later); it asserts successful FPM/OPcache collection and real worker/queue/cache series while awake, failed collection without stale worker/cache values while asleep, then successful recovery. Exporter probes must not refresh idle time or request wake. The separate process inherits the shared temporary probe directory and is terminated after the test. Without an explicit binary, this extra coverage is not exercised. No exporter dependency is added to init. This does not validate the baseimage's bundled exporter release or cbox-init startup configuration.
The transport has passed this test with real FPM, but CRIU remains unverified.
The runner also includes a negative qualification case: a PHP fixture calls
fastcgi_finish_request() and continues working for five seconds. The client
receives its complete response, closes, and the proxy reports idle; an independent
FPM status query still observes the script executing. This is a deliberately
reproduced transport limitation, not a successful CRIU qualification. The test
now also supplies the native FPM quiescence guard to the coordinator: checkpoint
is refused while that script runs and admitted after the script finishes. The
checkpoint controller remains fake. Even a complete
FastCGI response does not prove PHP stopped executing. The normal OPcache probe
does not call this function, but application-controlled PHP startup/shutdown code
cannot be assumed absent from a shared FPM pool. A production solution must prove
PHP quiescence or isolate the observation execution before allowing checkpoint;
a connection-drain timeout alone cannot provide that proof. The exporter
also needs a probe script path visible to FPM, and its placement must preserve
the snapshot single-process-tree requirement. Do not enable an additional
managed exporter process in the snapshot workload until these integration
constraints and checkpoint/restore acceptance tests are resolved.
Cgroup CPU accounting (unreleased)
The metrics server also exports the kernel's cumulative cgroup v2 CPU usage:
cbox_init_cgroup_cpu_usage_seconds_total: CPU seconds for the rooted cgroup subtree, including child processes that have already exited.cbox_init_cgroup_cpu_accounting_up:1when the current scrape read valid accounting;0when unavailable. On failure the counter is omitted, not zeroed.
This collector requires /proc/self/cgroup to identify 0::/ and a unified
/sys/fs/cgroup mount. Cgroup v1, non-rooted namespaces and non-Linux hosts are
unsupported and report up=0; no host-tree path is guessed. The measured scope
is that mount's rooted cgroup, so only run this as container accounting when the
runtime provides a private cgroup namespace/mount for that container.
CPU percentage gauges remain useful for live process health, but are not a replacement for this cumulative counter. The counter resets when the cgroup is recreated and does not preserve accounting across CRIU migration to another cgroup. Billing consumers must bind samples to the verified workload/container identity, detect resets, retain their own durable cursors and handle missed final samples. This metric alone does not make sampling lossless or authorize charging.
One Endpoint: Federation and Embedded Engines
Since 3.2.0 the main /metrics response can carry the whole container's
story, so Prometheus scrapes one port:
- Embedded engines merge natively. The runtime PHP-FPM autotuner's
fpm_tune_*series always appear on the main endpoint whenfpm_tuneis enabled.fpm_tune.metrics_addrremains optional, for parity with the standalone tool. - Local exporters federate. Declare them under
global.metrics_federateand their exposition is appended to every scrape, with a short cache so heavy scraping does not multiply load:
global:
metrics_federate:
- name: fpm-exporter
url: http://127.0.0.1:9114/metrics
timeout: 2s # per-fetch (default)
cache_ttl: 5s # cached between fetches (default)
Each source contributes a health gauge; a source that is down degrades instead of failing the scrape:
cbox_init_federate_up{name="fpm-exporter"} 1
Rules and caveats:
- URLs must point at loopback (
127.0.0.1,::1,localhost) — federation merges exporters inside the container; it is not a proxy, and config validation rejects anything else. - Bodies over 8 MiB and non-200 responses count as down.
- Metric names must not collide across sources — federation appends expositions verbatim and does not rewrite names.
- With federation enabled the endpoint always serves the plain-text exposition format (no content negotiation), because federated bodies are appended as-is.
A complete example lives in configs/examples/metrics-federate.yaml.
Available Metrics
Label Reference
Every series Cbox Init exposes under the cbox_init_ prefix, with its exact
label set. Selectors must use these keys: a selector on a label a metric does
not carry matches nothing.
| Metric | Type | Labels |
|---|---|---|
cbox_init_process_up |
Gauge | name, instance |
cbox_init_process_start_time_seconds |
Gauge | name, instance |
cbox_init_process_last_exit_code |
Gauge | name, instance |
cbox_init_process_restarts_total |
Counter | name, reason |
cbox_init_process_desired_scale |
Gauge | name |
cbox_init_process_current_scale |
Gauge | name |
cbox_init_health_check_status |
Gauge | name, type |
cbox_init_health_check_duration_seconds |
Histogram | name, type |
cbox_init_health_check_total |
Counter | name, type, status |
cbox_init_health_check_consecutive_fails |
Gauge | name |
cbox_init_process_cpu_percent |
Gauge | process, instance |
cbox_init_process_memory_bytes |
Gauge | process, instance, type |
cbox_init_process_memory_percent |
Gauge | process, instance |
cbox_init_process_threads |
Gauge | process, instance |
cbox_init_process_file_descriptors |
Gauge | process, instance |
cbox_init_resource_collection_errors_total |
Counter | process, instance |
cbox_init_resource_collection_duration_seconds |
Histogram | none |
cbox_init_hook_executions_total |
Counter | name, type, status |
cbox_init_hook_duration_seconds |
Histogram | name, type |
cbox_init_manager_process_count |
Gauge | none |
cbox_init_manager_start_time_seconds |
Gauge | none |
cbox_init_shutdown_duration_seconds |
Histogram | none |
cbox_init_build_info |
Gauge | version, go_version |
cbox_init_federate_up |
Gauge | name |
Label values:
nameis the process name from the config (php-fpm), except on the hook metrics, where it is the hook's name, and oncbox_init_federate_up, where it is the federated source's name.processis the process name from the config. Only the resource metrics use it.instanceis the instance ID,<process>-<index>counting from 0 (php-fpm-0,php-fpm-1, ...). See the note below on how Prometheus stores it.reason(restarts):crash,normal_exit,health_check,memory_limit.type: the health check type (tcp,http,exec) on health check metrics, the hook type (pre_start,post_start,pre_stop,post_stop) on hook metrics, andrssorvmsoncbox_init_process_memory_bytes.status:successorfailure.
name vs process
The process name sits under name on the lifecycle, health check and
scaling metrics, and under process on the resource metrics. The two
sets are not interchangeable: cbox_init_process_up{process="php-fpm"}
returns nothing. To combine them, rename one side with label_replace:
# CPU of instances that are currently up
cbox_init_process_cpu_percent
and on (instance, exported_instance, process)
label_replace(cbox_init_process_up == 1, "process", "$1", "name", "(.*)")
instance and exported_instance
Prometheus attaches its own instance label (the scrape target, e.g.
app:9090) to every series. With the default honor_labels: false, a
conflicting instance label in the scraped data is renamed to
exported_instance. So once scraped, the instance ID above is in
exported_instance and instance is the target:
# One instance of one container
cbox_init_process_up{instance="app:9090", name="php-fpm", exported_instance="php-fpm-0"}
The examples on this page use exported_instance in PromQL. With
honor_labels: true the instance ID stays in instance instead, and the
scrape target's instance is dropped for these series.
Series lifecycle
- A stopped or crashed instance keeps its series, with
cbox_init_process_upat0. That is the signal to alert on. - An instance removed by a scale-down has all of its per-instance series
deleted (
cbox_init_process_up,_start_time_seconds,_last_exit_code, and every resource metric), socbox_init_process_up == 0does not keep firing for an instance that was scaled away on purpose. The process-level series (restarts_total,desired_scale,current_scale, health checks) stay. This applies to a live scale-down and to a config reload or API edit that lowersscale. - A process removed from the config (through the API or a reload) has every one of its series deleted. So does a process that an edit or reload turns into a scheduled task: scheduled tasks export no process metrics (see Scheduled Tasks). A reload that fails and rolls back keeps the series, because the process comes back.
Process Lifecycle Metrics
cbox_init_process_up
Type: Gauge
Labels: name, instance
Description: Process status (1=running, 0=stopped)
# Query running instances of php-fpm
cbox_init_process_up{name="php-fpm"}
cbox_init_process_restarts_total
Type: Counter
Labels: name, reason
Description: Total number of process restarts by reason (crash, normal_exit, health_check, memory_limit)
# Total restarts for all processes
sum(cbox_init_process_restarts_total) by (name)
# Restarts due to health check failures
cbox_init_process_restarts_total{reason="health_check"}
cbox_init_process_start_time_seconds
Type: Gauge
Labels: name, instance
Description: Unix timestamp when process instance started
# Process uptime in seconds
time() - cbox_init_process_start_time_seconds
cbox_init_process_last_exit_code
Type: Gauge
Labels: name, instance
Description: Last exit code of process instance
# Non-zero exit codes (errors)
cbox_init_process_last_exit_code != 0
Health Check Metrics
cbox_init_health_check_status
Type: Gauge
Labels: name, type
Description: Health check status (1=healthy, 0=unhealthy)
# Unhealthy processes
cbox_init_health_check_status == 0
cbox_init_health_check_duration_seconds
Type: Histogram
Labels: name, type
Description: Health check duration in seconds
# 95th percentile health check latency
histogram_quantile(0.95,
sum(rate(cbox_init_health_check_duration_seconds_bucket[5m])) by (le, name)
)
cbox_init_health_check_total
Type: Counter
Labels: name, type, status
Description: Total number of health checks performed
# Health check failure rate
rate(cbox_init_health_check_total{status="failure"}[5m])
cbox_init_health_check_consecutive_fails
Type: Gauge
Labels: name
Description: Current consecutive health check failures
# Processes with multiple consecutive failures
cbox_init_health_check_consecutive_fails > 1
Scaling Metrics
cbox_init_process_desired_scale
Type: Gauge
Labels: name
Description: Desired number of process instances
# Desired scale configuration
cbox_init_process_desired_scale
cbox_init_process_current_scale
Type: Gauge
Labels: name
Description: Current number of running instances
# Scale drift (actual vs desired)
cbox_init_process_current_scale - cbox_init_process_desired_scale
Resource Metrics
Exposed when resource_metrics_enabled and metrics_enabled are both true.
These use process, not name, for the process name. Collection
details and more queries: Resource Monitoring.
cbox_init_process_cpu_percent
Type: Gauge
Labels: process, instance
Description: CPU usage in percent of one core (can exceed 100 on multi-core)
# Average CPU across php-fpm instances
avg(cbox_init_process_cpu_percent{process="php-fpm"})
cbox_init_process_memory_bytes
Type: Gauge
Labels: process, instance, type (rss, vms)
Description: Memory usage in bytes
# Resident memory per process
sum(cbox_init_process_memory_bytes{type="rss"}) by (process)
cbox_init_process_memory_percent
Type: Gauge
Labels: process, instance
Description: Memory usage as a percentage of total system memory
cbox_init_process_threads
Type: Gauge
Labels: process, instance
Description: Number of threads in the process
cbox_init_process_file_descriptors
Type: Gauge
Labels: process, instance
Description: Open file descriptors (Linux only; absent where unavailable)
cbox_init_resource_collection_errors_total
Type: Counter
Labels: process, instance
Description: Failed resource samples
cbox_init_resource_collection_duration_seconds
Type: Histogram Labels: none Description: Time taken by one resource collection pass
Hook Execution Metrics
cbox_init_hook_executions_total
Type: Counter
Labels: name (hook name), type, status
Description: Total hook executions by type and status
# Failed pre-start hooks
cbox_init_hook_executions_total{type="pre_start", status="failure"}
cbox_init_hook_duration_seconds
Type: Histogram
Labels: name (hook name), type
Description: Hook execution duration in seconds
# 99th percentile hook duration
histogram_quantile(0.99,
sum(rate(cbox_init_hook_duration_seconds_bucket[5m])) by (le, type)
)
Manager Metrics
cbox_init_manager_process_count
Type: Gauge Description: Total number of managed processes
# Total processes under management
cbox_init_manager_process_count
cbox_init_manager_start_time_seconds
Type: Gauge Description: Unix timestamp when manager started
# Manager uptime in seconds
time() - cbox_init_manager_start_time_seconds
cbox_init_shutdown_duration_seconds
Type: Histogram Labels: none Description: Duration of graceful shutdown in seconds
cbox_init_build_info
Type: Gauge
Labels: version, go_version
Description: Cbox Init build information
# Version information
cbox_init_build_info
Common Queries
Process Health Overview
# Running instances per process
sum(cbox_init_process_up) by (name)
# Count of processes with health check failures
count(cbox_init_health_check_status == 0) by (name)
Restart Monitoring
# Restart rate per minute
rate(cbox_init_process_restarts_total[1m])
# Processes restarting frequently (>5/hour)
sum(increase(cbox_init_process_restarts_total[1h])) by (name) > 5
Scale Monitoring
# Instances not matching desired scale
abs(cbox_init_process_current_scale - cbox_init_process_desired_scale) > 0
Hook Performance
# Slow hooks (>30s)
max(cbox_init_hook_duration_seconds) by (name, type) > 30
# Hook failure rate
rate(cbox_init_hook_executions_total{status="failure"}[5m])
Alerting Rules
Recommended Prometheus Alerts
groups:
- name: cbox_init
rules:
# Process down
- alert: ProcessDown
expr: cbox_init_process_up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Process {{ $labels.name }} ({{ $labels.exported_instance }}) on {{ $labels.instance }} is down"
# Frequent restarts
- alert: FrequentRestarts
expr: rate(cbox_init_process_restarts_total[5m]) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "Process {{ $labels.name }} restarting frequently"
# Health check failures
- alert: HealthCheckFailing
expr: cbox_init_health_check_status == 0
for: 2m
labels:
severity: warning
annotations:
summary: "Health check failing for {{ $labels.name }}"
# Scale drift
- alert: ScaleDrift
expr: abs(cbox_init_process_current_scale - cbox_init_process_desired_scale) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.name }} scale drift detected"
# Hook failures
- alert: HookFailures
expr: rate(cbox_init_hook_executions_total{status="failure"}[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Hook {{ $labels.name }} failing"
Grafana Dashboard
Sample Dashboard JSON
{
"dashboard": {
"title": "Cbox Init Overview",
"panels": [
{
"title": "Process Status",
"targets": [
{
"expr": "cbox_init_process_up"
}
]
},
{
"title": "Restart Rate",
"targets": [
{
"expr": "rate(cbox_init_process_restarts_total[5m])"
}
]
},
{
"title": "Health Check Status",
"targets": [
{
"expr": "cbox_init_health_check_status"
}
]
},
{
"title": "Scale Status",
"targets": [
{
"expr": "cbox_init_process_current_scale",
"legendFormat": "Current"
},
{
"expr": "cbox_init_process_desired_scale",
"legendFormat": "Desired"
}
]
}
]
}
}
Scraping Configuration
Prometheus Configuration
scrape_configs:
- job_name: 'cbox-init'
static_configs:
- targets: ['localhost:9090']
scrape_interval: 15s
Docker Compose Integration
services:
cbox-init:
image: cboxdk/init:latest
environment:
- CBOX_INIT_GLOBAL_METRICS_ENABLED=true
- CBOX_INIT_GLOBAL_METRICS_PORT=9090
ports:
- "9090:9090"
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9091:9090"
Native FPM check before checkpoint (unreleased)
CoordinatorOptions.Quiesce runs after proxy admission is closed, before
BeginCheckpoint. It receives a two-second context deadline. An error or expired
context keeps the workload running without consuming the failed-checkpoint budget.
After the check, the coordinator checks queued connections again and abandons
sleep if customer traffic arrived. Implementations must honor their context.
CheckFPMQuiescence uses the existing phpfpm native ReadStatus API on an explicit
list of pools. It rejects an empty inventory, unavailable status, mismatched pool
names, queued work, missing/duplicate worker identities, incomplete counts and any
state other than Idle plus exactly one Running native status request. This targets
the main-pool status listener; a separate status pool does not satisfy this model.
The check executes no PHP-info/OPcache probes and does not adjust native counters.
The shared status decoder rejects missing/null queue, active/idle/total and worker
fields, so unavailable data cannot become an apparent zero queue.
For a qualified Cbox PHP image with the single www pool, set
CBOX_SNAPSHOT_FPM_POOL=www together with a loopback CBOX_SNAPSHOT_UPSTREAM
such as 127.0.0.1:9000. Init uses the baseimage's native /fpm-status on
that listener. A configuration error keeps the workload serving through the
configured service proxy without registering for sleep. A busy or unavailable
status endpoint defers checkpoint on each idle poll. Leave this setting absent
for non-PHP engines.
Production activation still requires an authoritative complete pool inventory, admission exclusively through the proxy, treatment of detached/background processes and real CRIU acceptance. This helper cannot prove those conditions. No blanket claim of checkpoint safety follows from a successful single-pool status read.
The native API currently requires phpfpm's temporary public pseudo-version
v1.2.1-0.20261104150721-fd84ae99b5d3. Replace it with an approved tagged release
before publishing init/baseimages; no release is created by this change.