Self-hosted worker monitoring

The oz-agent-worker daemon exports OpenTelemetry metrics for worker health, task throughput, and capacity. Use these metrics with the <a href=https://oz.warp.dev>cloud agent dashboard to monitor self-hosted compute and cloud runs.

Key features

• Prometheus scrape — Expose a /metrics endpoint for Prometheus to scrape, with optional PodMonitor support for the Prometheus Operator.
• OTLP push — Push metrics to any OpenTelemetry-compatible collector (Grafana Alloy, Datadog Agent, New Relic, etc.).
• Standard configuration — Select an exporter with OpenTelemetry environment variables.
• Pre-seeded series — Metric series appear at startup before any tasks run.

How it works

The worker uses the OpenTelemetry autoexport package to select an exporter at runtime based on the OTEL_METRICS_EXPORTER environment variable. Supported values:

• prometheus — Starts an in-process HTTP server serving /metrics.
• otlp — Pushes metrics over OTLP (HTTP/protobuf by default).
• console — Writes metrics to stdout (useful for debugging).
• none — Disables metrics export entirely.

When OTEL_METRICS_EXPORTER is unset, autoexport uses OTLP. Set it to none to disable metrics.

All metrics carry resource attributes (service.name=oz-agent-worker, service.version, worker.id, worker.backend) so each worker process shows up as a distinct series in your monitoring system.

Enable Prometheus scrape

Set these environment variables before starting the worker to expose a Prometheus-compatible /metrics endpoint:

export OTEL_METRICS_EXPORTER=prometheus
export OTEL_EXPORTER_PROMETHEUS_HOST=0.0.0.0
export OTEL_EXPORTER_PROMETHEUS_PORT=9464
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

Verify the endpoint is serving metrics:

curl -s localhost:9464/metrics | grep oz_worker_

Expected outcome: You see oz_worker_connected, oz_worker_tasks_active, and other oz_worker_* metric families in the output.

Enable OTLP push

Set these environment variables to push metrics to an OpenTelemetry collector:

export OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.observability.svc:4318
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

The worker pushes metrics at the SDK's default interval. Configure the collector endpoint, protocol, and headers using standard OTLP exporter environment variables.

Helm chart configuration

The Helm chart includes built-in support for metrics. Enable metrics with metrics.enabled=true:

helm install oz-agent-worker ./charts/oz-agent-worker \
  --namespace warp-oz \
  --set worker.workerId=my-worker \
  --set image.tag=VERSION \
  --set metrics.enabled=true

With metrics.enabled=true and the default metrics.exporter=prometheus, the chart adds:

• A containerPort: metrics (default 9464) on the worker Deployment.
• The OTEL_METRICS_EXPORTER, OTEL_EXPORTER_PROMETHEUS_HOST, and OTEL_EXPORTER_PROMETHEUS_PORT environment variables.
• A namespace-scoped Service named <release>-oz-agent-worker-metrics with prometheus.io/scrape annotations.
• Optionally, a PodMonitor (metrics.podMonitor.create=true) for clusters using the Prometheus Operator.

Helm values

Core:

• metrics.enabled — Enable metrics export. Defaults to false.
• metrics.exporter — Exporter type: prometheus (default), otlp, console, or none.
• metrics.port — Port for the Prometheus exporter. Defaults to 9464. Ignored for otlp/console.
• metrics.extraEnv — Extra environment variables for the worker container (e.g., OTEL_EXPORTER_OTLP_ENDPOINT).

Service (Prometheus scrape):

• metrics.service.create — Create a metrics Service. Defaults to true.
• metrics.service.type — Service type. Defaults to ClusterIP.
• metrics.service.annotations — Annotations on the Service. Defaults include prometheus.io/scrape: "true".

PodMonitor (Prometheus Operator):

• metrics.podMonitor.create — Create a PodMonitor. Defaults to false (avoids requiring monitoring.coreos.com CRDs).
• metrics.podMonitor.interval — Scrape interval. Defaults to 30s.
• metrics.podMonitor.scrapeTimeout — Scrape timeout. Defaults to 10s.
• metrics.podMonitor.additionalLabels — Extra labels on the PodMonitor resource.

OTLP push via Helm

To push metrics to an OTLP collector instead of exposing a Prometheus endpoint, set metrics.exporter=otlp and forward the endpoint via metrics.extraEnv:

metrics:
  enabled: true
  exporter: otlp
  extraEnv:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector.observability.svc:4318

Metric catalog

All metrics use the oz_worker_ prefix. Each worker process with metrics enabled emits a distinct set of series, identified by the resource attributes service.name, service.version, worker.id, and worker.backend.

• oz_worker_connected (gauge) — 1 while the worker has an active WebSocket connection to the Automation Platform's backend, 0 otherwise.
• oz_worker_tasks_active (gauge / UpDownCounter) — Tasks currently executing on this worker.
• oz_worker_tasks_max_concurrent (gauge) — Configured concurrency limit (0 means unlimited).
• oz_worker_tasks_claimed_total (counter) — Total tasks accepted since process start.
• oz_worker_tasks_rejected_total{reason} (counter) — Tasks the worker declined (e.g., reason="at_capacity").
• oz_worker_tasks_completed_total{result} (counter) — Completed tasks labeled result="succeeded", result="failed", result="cancelled", or result="dispatched".
• oz_worker_task_duration_seconds{result} (histogram) — Wall-clock task duration on the worker, labeled by result.
• oz_worker_task_failures_total{phase,reason} (counter) — Task failures classified by execution phase and a bounded reason such as image_pull, unschedulable, container_oom, or evicted.
• oz_worker_websocket_reconnects_total{reason} (counter) — WebSocket reconnect attempts (e.g., reason="dial_failed", reason="remote_close"). Spikes indicate flapping workers.
• oz_worker_info{version,backend,worker_id} (gauge, constant 1) — Build and runtime metadata. Useful for joining other series by labels.

Sample PromQL queries

• Workers available:

  sum(oz_worker_connected)

• Workers active (running at least one task):

  count(oz_worker_tasks_active > 0)

• Fleet saturation:

  sum(oz_worker_tasks_active) / sum(oz_worker_tasks_max_concurrent > 0)

  This ratio is only meaningful when every worker has a non-zero oz_worker_tasks_max_concurrent. Workers configured with 0 (unlimited) are excluded from the denominator, which can make the saturation result look misleadingly high or undefined for fleets that mix bounded and unlimited workers.

• Task success rate (5-minute window):

  sum(rate(oz_worker_tasks_completed_total{result="succeeded"}[5m]))
  / sum(rate(oz_worker_tasks_completed_total[5m]))

• Task duration p95:

  histogram_quantile(0.95, sum by (le) (rate(oz_worker_task_duration_seconds_bucket[5m])))

• Failure rate:

  sum(rate(oz_worker_tasks_completed_total{result="failed"}[5m]))

• Failure modes:

  sum by (phase, reason) (rate(oz_worker_task_failures_total[5m]))

• Reconnect storms (alert threshold):

  sum(rate(oz_worker_websocket_reconnects_total[5m])) > 0.1

Disabling metrics

Set OTEL_METRICS_EXPORTER=none to disable metrics export:

export OTEL_METRICS_EXPORTER=none
oz-agent-worker --api-key "$WARP_API_KEY" --worker-id "my-worker"

The Helm chart sets the same override when metrics are disabled:

metrics:
  enabled: false

Related pages

• Self-hosting overview — Architecture, decision guide, and Enterprise requirements.
• Self-hosted worker reference — CLI flags, config file schema, and metrics environment variables.
• Managed: Kubernetes — Helm chart deployment, including metrics values.
• Troubleshooting — Diagnostics for metrics issues and other common problems.
• Security and networking — Network egress and data boundaries.