Skip to content

Prometheus Metrics

Prometheus Metrics

Comprehensive Prometheus metrics for monitoring Cbox Init and managed processes.

Configuration

Enable metrics in your cbox-init.yaml:

global:
  metrics_enabled: true
  metrics_port: 9090
  metrics_path: /metrics

Non-waking FPM observations (unreleased)

PHP baseimages already provide FPM and OPcache metrics through fpm-exporter. Reuse that exporter; cbox-init does not implement a second FPM collector. For snapshot workloads, the optional CBOX_SNAPSHOT_OBSERVE_LISTEN address provides a monitoring connection to CBOX_SNAPSHOT_UPSTREAM without requesting restore or extending the customer idle timer. For example, with the ordinary service proxy configured on 127.0.0.1:9001, use 127.0.0.1:9002 for observations and point the exporter's FastCGI connection at that observation port.

Only explicit loopback IP addresses are accepted. This is privileged raw upstream access, not HTTP, and must never be exposed through ingress or a Service. Processes sharing the pod network can reach it. An admitted probe pins the workload against checkpoint while its connection is open, for at most five seconds. Probes arriving during sleep/checkpoint are disconnected without wake. Missing samples must be represented as unavailable, not as zero or disabled OPcache. All monitoring traffic must use this path; direct upstream connections bypass its coordination.

Run make test-e2e-fpm-observation to verify real FPM status and enabled OPcache through the observation proxy using the Cbox PHP baseimage. The test checks that probes leave the customer idle clock untouched, are refused while asleep without wake, and resume after the awake transition. It prints the tested image ID; set CBOX_TEST_PHP_IMAGE to select another image or immutable digest. Its lifecycle control is fake: it does not perform a CRIU checkpoint or restore.

Verified locally on 2026-11-04 with race detection:

  • PHP 8.4 Bookworm, image ID sha256:9e0d7787d5646436dc3839090ba6fd18a25da973e9e1db4cfa06f513d56900ba.
  • PHP 8.5 Bookworm rootless, image ID sha256:b4d7228dc08320fa8464f961725ae06f0216dcf8f547e8c7e0d9430dede63c20.

These are local Docker image IDs, not registry manifest digests. By default the test runs this checkout's proxy with the existing phpfpm dependency. To include the actual exporter process, build the selected fpm-exporter checkout separately and pass its absolute executable path:

CBOX_TEST_FPM_EXPORTER_BINARY=/absolute/path/to/fpm-exporter make test-e2e-fpm-observation

The runner prints the binary SHA-256. This requires the unreleased exporter observation-quality changes (9e2b980 or later); it asserts successful FPM/OPcache collection and real worker/queue/cache series while awake, failed collection without stale worker/cache values while asleep, then successful recovery. Exporter probes must not refresh idle time or request wake. The separate process inherits the shared temporary probe directory and is terminated after the test. Without an explicit binary, this extra coverage is not exercised. No exporter dependency is added to init. This does not validate the baseimage's bundled exporter release or cbox-init startup configuration.

The transport has passed this test with real FPM, but CRIU remains unverified. The runner also includes a negative qualification case: a PHP fixture calls fastcgi_finish_request() and continues working for five seconds. The client receives its complete response, closes, and the proxy reports idle; an independent FPM status query still observes the script executing. This is a deliberately reproduced transport limitation, not a successful CRIU qualification. The test now also supplies the native FPM quiescence guard to the coordinator: checkpoint is refused while that script runs and admitted after the script finishes. The checkpoint controller remains fake. Even a complete FastCGI response does not prove PHP stopped executing. The normal OPcache probe does not call this function, but application-controlled PHP startup/shutdown code cannot be assumed absent from a shared FPM pool. A production solution must prove PHP quiescence or isolate the observation execution before allowing checkpoint; a connection-drain timeout alone cannot provide that proof. The exporter also needs a probe script path visible to FPM, and its placement must preserve the snapshot single-process-tree requirement. Do not enable an additional managed exporter process in the snapshot workload until these integration constraints and checkpoint/restore acceptance tests are resolved.

Cgroup CPU accounting (unreleased)

The metrics server also exports the kernel's cumulative cgroup v2 CPU usage:

  • cbox_init_cgroup_cpu_usage_seconds_total: CPU seconds for the rooted cgroup subtree, including child processes that have already exited.
  • cbox_init_cgroup_cpu_accounting_up: 1 when the current scrape read valid accounting; 0 when unavailable. On failure the counter is omitted, not zeroed.

This collector requires /proc/self/cgroup to identify 0::/ and a unified /sys/fs/cgroup mount. Cgroup v1, non-rooted namespaces and non-Linux hosts are unsupported and report up=0; no host-tree path is guessed. The measured scope is that mount's rooted cgroup, so only run this as container accounting when the runtime provides a private cgroup namespace/mount for that container.

CPU percentage gauges remain useful for live process health, but are not a replacement for this cumulative counter. The counter resets when the cgroup is recreated and does not preserve accounting across CRIU migration to another cgroup. Billing consumers must bind samples to the verified workload/container identity, detect resets, retain their own durable cursors and handle missed final samples. This metric alone does not make sampling lossless or authorize charging.

One Endpoint: Federation and Embedded Engines

Since 3.2.0 the main /metrics response can carry the whole container's story, so Prometheus scrapes one port:

  • Embedded engines merge natively. The runtime PHP-FPM autotuner's fpm_tune_* series always appear on the main endpoint when fpm_tune is enabled. fpm_tune.metrics_addr remains optional, for parity with the standalone tool.
  • Local exporters federate. Declare them under global.metrics_federate and their exposition is appended to every scrape, with a short cache so heavy scraping does not multiply load:
global:
  metrics_federate:
    - name: fpm-exporter
      url: http://127.0.0.1:9114/metrics
      timeout: 2s     # per-fetch (default)
      cache_ttl: 5s   # cached between fetches (default)

Each source contributes a health gauge; a source that is down degrades instead of failing the scrape:

cbox_init_federate_up{name="fpm-exporter"} 1

Rules and caveats:

  • URLs must point at loopback (127.0.0.1, ::1, localhost) — federation merges exporters inside the container; it is not a proxy, and config validation rejects anything else.
  • Bodies over 8 MiB and non-200 responses count as down.
  • Metric names must not collide across sources — federation appends expositions verbatim and does not rewrite names.
  • With federation enabled the endpoint always serves the plain-text exposition format (no content negotiation), because federated bodies are appended as-is.

A complete example lives in configs/examples/metrics-federate.yaml.

Available Metrics

Label Reference

Every series Cbox Init exposes under the cbox_init_ prefix, with its exact label set. Selectors must use these keys: a selector on a label a metric does not carry matches nothing.

Metric Type Labels
cbox_init_process_up Gauge name, instance
cbox_init_process_start_time_seconds Gauge name, instance
cbox_init_process_last_exit_code Gauge name, instance
cbox_init_process_restarts_total Counter name, reason
cbox_init_process_desired_scale Gauge name
cbox_init_process_current_scale Gauge name
cbox_init_health_check_status Gauge name, type
cbox_init_health_check_duration_seconds Histogram name, type
cbox_init_health_check_total Counter name, type, status
cbox_init_health_check_consecutive_fails Gauge name
cbox_init_process_cpu_percent Gauge process, instance
cbox_init_process_memory_bytes Gauge process, instance, type
cbox_init_process_memory_percent Gauge process, instance
cbox_init_process_threads Gauge process, instance
cbox_init_process_file_descriptors Gauge process, instance
cbox_init_resource_collection_errors_total Counter process, instance
cbox_init_resource_collection_duration_seconds Histogram none
cbox_init_hook_executions_total Counter name, type, status
cbox_init_hook_duration_seconds Histogram name, type
cbox_init_manager_process_count Gauge none
cbox_init_manager_start_time_seconds Gauge none
cbox_init_shutdown_duration_seconds Histogram none
cbox_init_build_info Gauge version, go_version
cbox_init_federate_up Gauge name

Label values:

  • name is the process name from the config (php-fpm), except on the hook metrics, where it is the hook's name, and on cbox_init_federate_up, where it is the federated source's name.
  • process is the process name from the config. Only the resource metrics use it.
  • instance is the instance ID, <process>-<index> counting from 0 (php-fpm-0, php-fpm-1, ...). See the note below on how Prometheus stores it.
  • reason (restarts): crash, normal_exit, health_check, memory_limit.
  • type: the health check type (tcp, http, exec) on health check metrics, the hook type (pre_start, post_start, pre_stop, post_stop) on hook metrics, and rss or vms on cbox_init_process_memory_bytes.
  • status: success or failure.

name vs process

The process name sits under name on the lifecycle, health check and scaling metrics, and under process on the resource metrics. The two sets are not interchangeable: cbox_init_process_up{process="php-fpm"} returns nothing. To combine them, rename one side with label_replace:

# CPU of instances that are currently up
cbox_init_process_cpu_percent
  and on (instance, exported_instance, process)
label_replace(cbox_init_process_up == 1, "process", "$1", "name", "(.*)")

instance and exported_instance

Prometheus attaches its own instance label (the scrape target, e.g. app:9090) to every series. With the default honor_labels: false, a conflicting instance label in the scraped data is renamed to exported_instance. So once scraped, the instance ID above is in exported_instance and instance is the target:

# One instance of one container
cbox_init_process_up{instance="app:9090", name="php-fpm", exported_instance="php-fpm-0"}

The examples on this page use exported_instance in PromQL. With honor_labels: true the instance ID stays in instance instead, and the scrape target's instance is dropped for these series.

Series lifecycle

  • A stopped or crashed instance keeps its series, with cbox_init_process_up at 0. That is the signal to alert on.
  • An instance removed by a scale-down has all of its per-instance series deleted (cbox_init_process_up, _start_time_seconds, _last_exit_code, and every resource metric), so cbox_init_process_up == 0 does not keep firing for an instance that was scaled away on purpose. The process-level series (restarts_total, desired_scale, current_scale, health checks) stay. This applies to a live scale-down and to a config reload or API edit that lowers scale.
  • A process removed from the config (through the API or a reload) has every one of its series deleted. So does a process that an edit or reload turns into a scheduled task: scheduled tasks export no process metrics (see Scheduled Tasks). A reload that fails and rolls back keeps the series, because the process comes back.

Process Lifecycle Metrics

cbox_init_process_up

Type: Gauge Labels: name, instance Description: Process status (1=running, 0=stopped)

# Query running instances of php-fpm
cbox_init_process_up{name="php-fpm"}

cbox_init_process_restarts_total

Type: Counter Labels: name, reason Description: Total number of process restarts by reason (crash, normal_exit, health_check, memory_limit)

# Total restarts for all processes
sum(cbox_init_process_restarts_total) by (name)

# Restarts due to health check failures
cbox_init_process_restarts_total{reason="health_check"}

cbox_init_process_start_time_seconds

Type: Gauge Labels: name, instance Description: Unix timestamp when process instance started

# Process uptime in seconds
time() - cbox_init_process_start_time_seconds

cbox_init_process_last_exit_code

Type: Gauge Labels: name, instance Description: Last exit code of process instance

# Non-zero exit codes (errors)
cbox_init_process_last_exit_code != 0

Health Check Metrics

cbox_init_health_check_status

Type: Gauge Labels: name, type Description: Health check status (1=healthy, 0=unhealthy)

# Unhealthy processes
cbox_init_health_check_status == 0

cbox_init_health_check_duration_seconds

Type: Histogram Labels: name, type Description: Health check duration in seconds

# 95th percentile health check latency
histogram_quantile(0.95,
  sum(rate(cbox_init_health_check_duration_seconds_bucket[5m])) by (le, name)
)

cbox_init_health_check_total

Type: Counter Labels: name, type, status Description: Total number of health checks performed

# Health check failure rate
rate(cbox_init_health_check_total{status="failure"}[5m])

cbox_init_health_check_consecutive_fails

Type: Gauge Labels: name Description: Current consecutive health check failures

# Processes with multiple consecutive failures
cbox_init_health_check_consecutive_fails > 1

Scaling Metrics

cbox_init_process_desired_scale

Type: Gauge Labels: name Description: Desired number of process instances

# Desired scale configuration
cbox_init_process_desired_scale

cbox_init_process_current_scale

Type: Gauge Labels: name Description: Current number of running instances

# Scale drift (actual vs desired)
cbox_init_process_current_scale - cbox_init_process_desired_scale

Resource Metrics

Exposed when resource_metrics_enabled and metrics_enabled are both true. These use process, not name, for the process name. Collection details and more queries: Resource Monitoring.

cbox_init_process_cpu_percent

Type: Gauge Labels: process, instance Description: CPU usage in percent of one core (can exceed 100 on multi-core)

# Average CPU across php-fpm instances
avg(cbox_init_process_cpu_percent{process="php-fpm"})

cbox_init_process_memory_bytes

Type: Gauge Labels: process, instance, type (rss, vms) Description: Memory usage in bytes

# Resident memory per process
sum(cbox_init_process_memory_bytes{type="rss"}) by (process)

cbox_init_process_memory_percent

Type: Gauge Labels: process, instance Description: Memory usage as a percentage of total system memory

cbox_init_process_threads

Type: Gauge Labels: process, instance Description: Number of threads in the process

cbox_init_process_file_descriptors

Type: Gauge Labels: process, instance Description: Open file descriptors (Linux only; absent where unavailable)

cbox_init_resource_collection_errors_total

Type: Counter Labels: process, instance Description: Failed resource samples

cbox_init_resource_collection_duration_seconds

Type: Histogram Labels: none Description: Time taken by one resource collection pass

Hook Execution Metrics

cbox_init_hook_executions_total

Type: Counter Labels: name (hook name), type, status Description: Total hook executions by type and status

# Failed pre-start hooks
cbox_init_hook_executions_total{type="pre_start", status="failure"}

cbox_init_hook_duration_seconds

Type: Histogram Labels: name (hook name), type Description: Hook execution duration in seconds

# 99th percentile hook duration
histogram_quantile(0.99,
  sum(rate(cbox_init_hook_duration_seconds_bucket[5m])) by (le, type)
)

Manager Metrics

cbox_init_manager_process_count

Type: Gauge Description: Total number of managed processes

# Total processes under management
cbox_init_manager_process_count

cbox_init_manager_start_time_seconds

Type: Gauge Description: Unix timestamp when manager started

# Manager uptime in seconds
time() - cbox_init_manager_start_time_seconds

cbox_init_shutdown_duration_seconds

Type: Histogram Labels: none Description: Duration of graceful shutdown in seconds

cbox_init_build_info

Type: Gauge Labels: version, go_version Description: Cbox Init build information

# Version information
cbox_init_build_info

Common Queries

Process Health Overview

# Running instances per process
sum(cbox_init_process_up) by (name)

# Count of processes with health check failures
count(cbox_init_health_check_status == 0) by (name)

Restart Monitoring

# Restart rate per minute
rate(cbox_init_process_restarts_total[1m])

# Processes restarting frequently (>5/hour)
sum(increase(cbox_init_process_restarts_total[1h])) by (name) > 5

Scale Monitoring

# Instances not matching desired scale
abs(cbox_init_process_current_scale - cbox_init_process_desired_scale) > 0

Hook Performance

# Slow hooks (>30s)
max(cbox_init_hook_duration_seconds) by (name, type) > 30

# Hook failure rate
rate(cbox_init_hook_executions_total{status="failure"}[5m])

Alerting Rules

groups:
  - name: cbox_init
    rules:
      # Process down
      - alert: ProcessDown
        expr: cbox_init_process_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Process {{ $labels.name }} ({{ $labels.exported_instance }}) on {{ $labels.instance }} is down"

      # Frequent restarts
      - alert: FrequentRestarts
        expr: rate(cbox_init_process_restarts_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Process {{ $labels.name }} restarting frequently"

      # Health check failures
      - alert: HealthCheckFailing
        expr: cbox_init_health_check_status == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Health check failing for {{ $labels.name }}"

      # Scale drift
      - alert: ScaleDrift
        expr: abs(cbox_init_process_current_scale - cbox_init_process_desired_scale) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} scale drift detected"

      # Hook failures
      - alert: HookFailures
        expr: rate(cbox_init_hook_executions_total{status="failure"}[5m]) > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Hook {{ $labels.name }} failing"

Grafana Dashboard

Sample Dashboard JSON

{
  "dashboard": {
    "title": "Cbox Init Overview",
    "panels": [
      {
        "title": "Process Status",
        "targets": [
          {
            "expr": "cbox_init_process_up"
          }
        ]
      },
      {
        "title": "Restart Rate",
        "targets": [
          {
            "expr": "rate(cbox_init_process_restarts_total[5m])"
          }
        ]
      },
      {
        "title": "Health Check Status",
        "targets": [
          {
            "expr": "cbox_init_health_check_status"
          }
        ]
      },
      {
        "title": "Scale Status",
        "targets": [
          {
            "expr": "cbox_init_process_current_scale",
            "legendFormat": "Current"
          },
          {
            "expr": "cbox_init_process_desired_scale",
            "legendFormat": "Desired"
          }
        ]
      }
    ]
  }
}

Scraping Configuration

Prometheus Configuration

scrape_configs:
  - job_name: 'cbox-init'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 15s

Docker Compose Integration

services:
  cbox-init:
    image: cboxdk/init:latest
    environment:
      - CBOX_INIT_GLOBAL_METRICS_ENABLED=true
      - CBOX_INIT_GLOBAL_METRICS_PORT=9090
    ports:
      - "9090:9090"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9091:9090"

Native FPM check before checkpoint (unreleased)

CoordinatorOptions.Quiesce runs after proxy admission is closed, before BeginCheckpoint. It receives a two-second context deadline. An error or expired context keeps the workload running without consuming the failed-checkpoint budget. After the check, the coordinator checks queued connections again and abandons sleep if customer traffic arrived. Implementations must honor their context.

CheckFPMQuiescence uses the existing phpfpm native ReadStatus API on an explicit list of pools. It rejects an empty inventory, unavailable status, mismatched pool names, queued work, missing/duplicate worker identities, incomplete counts and any state other than Idle plus exactly one Running native status request. This targets the main-pool status listener; a separate status pool does not satisfy this model. The check executes no PHP-info/OPcache probes and does not adjust native counters. The shared status decoder rejects missing/null queue, active/idle/total and worker fields, so unavailable data cannot become an apparent zero queue.

For a qualified Cbox PHP image with the single www pool, set CBOX_SNAPSHOT_FPM_POOL=www together with a loopback CBOX_SNAPSHOT_UPSTREAM such as 127.0.0.1:9000. Init uses the baseimage's native /fpm-status on that listener. A configuration error keeps the workload serving through the configured service proxy without registering for sleep. A busy or unavailable status endpoint defers checkpoint on each idle poll. Leave this setting absent for non-PHP engines.

Production activation still requires an authoritative complete pool inventory, admission exclusively through the proxy, treatment of detached/background processes and real CRIU acceptance. This helper cannot prove those conditions. No blanket claim of checkpoint safety follows from a successful single-pool status read.

The native API currently requires phpfpm's temporary public pseudo-version v1.2.1-0.20261104150721-fd84ae99b5d3. Replace it with an approved tagged release before publishing init/baseimages; no release is created by this change.