Observability

The Signal Hierarchy

Three signals, three roles. Operators tend to want all three but for very different reasons.

Signed receipts are the primary audit record. Traces and metrics are operational telemetry. Deploy receipts when you need a durable audit record.

OTel Collector Wiring

Chio components emit OTLP gRPC traces and Prometheus metrics directly. A standard OpenTelemetry Collector is the right aggregation point for everything except receipts. Receipts go to the receipt store first; the SIEM exporter or the OTel receipt exporter pulls from there.

The collector config below is the minimum that powers the five shipped dashboards. Receivers pick up OTLP traces; processors strip high-cardinality attributes before they hit Prometheus; exporters fan out to Tempo, Loki, and (optionally) the receipt exporter.

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

  # Strip high-cardinality attributes from anything bound for Prometheus.
  # These attributes are safe on spans and logs; they are not safe as
  # Prometheus label values.
  attributes/strip-cardinality:
    actions:
      - key: gen_ai.tool.call.id
        action: delete
      - key: chio.receipt.id
        action: delete
      - key: chio.replay.run_id
        action: delete
      - key: chio.tenant.id
        action: delete

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

use chio_kernel::otel::{ATTR_CHIO_RECEIPT_ID, ATTR_GEN_AI_TOOL_CALL_ID};

pub const ATTR_CHIO_REPLAY_RUN_ID: &str = "chio.replay.run_id";
pub const ATTR_CHIO_TENANT_ID: &str = "chio.tenant.id";

pub const PROMETHEUS_DENIED_ATTRIBUTES: [&str; 4] = [
    ATTR_GEN_AI_TOOL_CALL_ID,
    ATTR_CHIO_RECEIPT_ID,
    ATTR_CHIO_REPLAY_RUN_ID,
    ATTR_CHIO_TENANT_ID,
];

The exporter calls strip_denied_attributes on every span before signing the receipt body (denylist.rs:21-27), so the canonicalized form is stable across re-export. Ingest the same span twice and you get the same receipt body bytes.

The Receipt Exporter Crate

chio-otel-receipt-exporter is the bridge from OTLP trace ingress to the chio receipt store. Each accepted span gets canonicalized, signed, and appended to the configured ReceiptStore. The sanitizer strips the same denied attributes before signing so the canonical form is stable across re-export.

use chio_otel_receipt_exporter::{ReceiptStoreSink, ReceiptStoreSinkConfig};

let config = ReceiptStoreSinkConfig {
    signing_keypair,                       // Ed25519 keypair
    policy_hash: "otel-receipt-exporter".into(),
    default_capability_id: "otel-ingress".into(),
    default_tool_server: "otel-collector".into(),
    default_tool_name: "gen_ai.tool.call".into(),
    tenant_id: Some("acme-prod".into()),
};

let sink = ReceiptStoreSink::new(receipt_store, config);
let summary = sink.export_traces(&otlp_export)?;
println!("appended {} receipts", summary.appended_receipts);

The sink is single-purpose: it takes an OtlpGrpcTraceExport, produces one ChioReceipt per span (with ToolCallAction carrying the sanitized attributes), and appends each receipt through the same ReceiptStore trait the kernel uses. Span identity carries through: trace_id and span_id are validated before signing.

Trace Propagation

Chio respects W3C Trace Context. When the upstream caller passes a traceparent header, the kernel joins that trace and emits its spans as children. The receipt then carries provenance.otel.trace_id and provenance.otel.span_id as fields, which is what makes the Tempo and Jaeger dashboards able to pivot from a receipt to a trace.

That linkage is the M10 attribute lock. The required structured fields on every span and log line are:

• chio.receipt.id
• chio.tenant.id
• chio.policy.ref
• chio.verdict
• chio.tee.mode
• chio.deny.reason
• chio.guard.outcome
• provenance.otel.trace_id
• provenance.otel.span_id
• redaction_pass_id
• redaction_elapsed_micros

Metrics Scrape Config

Chio exposes Prometheus metrics over an HTTP /metrics endpoint. The WASM guard runtime publishes a fixed set of families documented in crates/guards/chio-wasm-guards/src/metrics.rs:

• chio_guard_eval_duration_seconds histogram, labels guard_id, verdict.
• chio_guard_fuel_consumed_total counter, label guard_id.
• chio_guard_verdict_total counter, labels guard_id, verdict.
• chio_guard_deny_total counter, labels guard_id, reason_class.
• chio_guard_reload_total counter, labels guard_id, outcome.
• chio_guard_host_call_duration_seconds histogram, labels guard_id, host_fn.
• chio_guard_module_bytes gauge, labels guard_id, epoch.

Total label cardinality is capped at MAX_GUARD_METRIC_CARDINALITY = 1024; registering more guards than that fails closed with E_GUARD_METRIC_CARDINALITY_EXCEEDED.

# prometheus.yml
scrape_configs:
  - job_name: 'chio'
    scrape_interval: 15s
    metrics_path: /metrics
    static_configs:
      - targets:
          - chio-edge:9091
          - chio-tee:9091
          - chio-control:9091

The Documented Dashboards

Five Grafana dashboard JSON files are documented in deploy/dashboards/README.md. All five expect the M10 attribute lock and use Grafana datasource variables (DS_LOKI, DS_TEMPO, DS_JAEGER) so import works across stacks. The one-command import script lives at the top of that README. The deploy/dashboards/ tree also carries two dashboards outside the documented set — grafana/chio-perf.json and chio-pheromone-relay-observability.json — that the README does not yet list.

loki/chio-tee.json

Title: Chio TEE Loki receipt view. Targets Loki. Four panels: TEE records by verdict, TEE mode and verdict density heatmap, raw log stream, and a receipt-to-trace lookup table (top 50 by receipt id, with policy ref, verdict, deny reason). Filterable by receipt id, tenant id, trace id, verdict, and TEE mode.

loki/verdict-drift.json

Title: Chio replay verdict drift. Targets Loki. Filters chio-replay logs to event="verdict_drift" and renders a heatmap of from_verdict × to_verdict, a stacked time series of reason_delta, a top-100 drift records table, and a raw drift log stream.

jaeger/receipt-span-lookup.json

Title: Chio Jaeger receipt span lookup. Targets Jaeger. Pivots from receipt id, trace id, span id, tenant id, and tool name to traces. Four panels: trace search by tags, table of receipt and span lookups, traces-by-verdict time series, and trace-duration time series.

tempo/span-timeline.json

Title: Chio Tempo span timeline. Targets Tempo. The primary panel runs the TraceQL query { span.chio.receipt.id = "$receipt_id" } to fetch every trace that touched a given receipt. Additional panels: a span rows table, a GenAI tool-call rate time series, and a span-duration-by-verdict line chart.

tempo/redaction-latency.json

Title: Chio Tempo redaction latency. Targets Tempo. Tracks redaction-pass latency by redaction_pass_id and chio.guard.outcome. Four panels: p95 latency, latency distribution histogram, redaction pass spans table, and span rate.

Alert Rules

Chio ships a rule pack in deploy/prometheus/ (the “T1.5 SRE rule pack” per deploy/prometheus/README.md), not a collection of ad hoc alerts. Two files define the rules: chio-recording-rules.yml precomputes the error-ratio and p95 series the SLOs burn against, and chio-alert-rules.yml layers multi-window burn-rate and missing-data alerts on top. Every alert carries severity, notification_route, and opsgenie labels — the handoff contract to the chio-siem / chio-wall PagerDuty and OpsGenie dispatch path.

Recording rules

Burn-rate alerts fire on ratios, so the ratios are recorded first.

groups:
  - name: chio-sre-recording
    interval: 30s
    rules:
      - record: chio:receipt_write_error_ratio_1h
        expr: |
          sum(rate(chio_receipt_write_total{outcome="error"}[1h]))
          /
          clamp_min(sum(rate(chio_receipt_write_total[1h])), 1)

      - record: chio:receipt_write_error_ratio_6h
        expr: |
          sum(rate(chio_receipt_write_total{outcome="error"}[6h]))
          /
          clamp_min(sum(rate(chio_receipt_write_total[6h])), 1)

      - record: chio:sidecar_error_ratio_1h
        expr: |
          sum(rate(chio_sidecar_requests_total{outcome!="success"}[1h]))
          /
          clamp_min(sum(rate(chio_sidecar_requests_total[1h])), 1)

clamp_min(..., 1) in the denominator keeps a quiet window from dividing by zero and reporting a false 100% error ratio. The pack records the same 1h/6h pair for alert dispatch and SOC export, plus p95 latency series for kernel decisions, guard evaluation, anchor rounds, and federation hops.

Burn-rate SLO alerts

Each SLO uses a two-window burn-rate: the short window catches a fast burn, the long window suppresses flapping. Receipt writes hold a 0.1% error budget.

- alert: ChioReceiptWriteErrorBudgetBurn1h
  expr: |
    chio:receipt_write_error_ratio_1h > (14.4 * 0.001)
    and
    chio:receipt_write_error_ratio_6h > (6 * 0.001)
  for: 2m
  labels:
    severity: p1
    notification_route: pagerduty
    opsgenie: "true"
    slo: receipt-write
  annotations:
    summary: Chio receipt writes are burning error budget
    runbook: docs/operator-runbook/slo.md

The same shape covers ChioSidecarErrorBudgetBurn1h (0.5% budget, p2) and ChioAlertDispatchErrorBudgetBurn1h / ChioSocExportErrorBudgetBurn1h (1% budget each).

Missing-data and zero-tolerance alerts

A silent exporter looks identical to a healthy one on a ratio, so every SLO pairs its burn-rate alert with an absent_over_time watchdog. Some conditions carry no budget at all and page on the first occurrence.

- alert: ChioReceiptWriteMetricsMissing
  expr: absent_over_time(chio_receipt_write_total[10m])
  for: 2m
  labels:
    severity: p1
    notification_route: pagerduty
    opsgenie: "true"
    slo: receipt-write

- alert: ChioFailOpenSuspected
  expr: increase(chio_fail_open_suspected_total[5m]) > 0
  for: 0m
  labels:
    severity: p0
    notification_route: pagerduty
    opsgenie: "true"
    slo: fail-closed

- alert: ChioRevocationLagHigh
  expr: |
    histogram_quantile(
      0.95,
      sum by (le) (rate(chio_capability_revocation_lag_seconds_bucket[5m]))
    ) > 30
  for: 10m
  labels:
    severity: p1
    notification_route: pagerduty
    opsgenie: "true"
    slo: revocation

The full pack also carries ChioDispatchFailure, ChioReceiptCheckpointStale (fires when checkpointing stalls past 24 hours), and a *MetricsMissing counterpart for each burn-rate and zero-tolerance alert.

SIEM Integration

For long-term retention and security workflow, receipts go to a SIEM rather than a metrics or trace backend. The exporter is independent of the kernel and uses a sequence cursor so it never loses data on retry. Read /docs/deployment/siem-export for the design, configuration, and OCSF field mapping.

Worked Example: docker-compose

A minimal stack that runs an OTel collector, Prometheus, Loki, Tempo, and Grafana, and imports the dashboards on startup. Adapt the Chio service definition to your deployment shape.

# docker-compose.yaml
services:
  chio-edge:
    image: chio-sidecar:latest
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
      OTEL_SERVICE_NAME: chio-edge
    ports:
      - "9091:9091"      # /metrics

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otel-collector.yaml:ro
    ports:
      - "4317:4317"      # OTLP gRPC
      - "4318:4318"      # OTLP HTTP

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml:ro

  grafana:
    image: grafana/grafana:latest
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    ports:
      - "3000:3000"
    volumes:
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/ds.yaml:ro

Once Grafana is up, the dashboard import is one command from the Chio repo root:

$ find deploy/dashboards -name '*.json' -print0 | \
    while IFS= read -r -d '' dashboard; do
      jq -n --argjson dashboard "$(cat "$dashboard")" \
        '{dashboard: $dashboard, overwrite: true}' \
      | curl -fsS -H 'Content-Type: application/json' \
          -X POST http://admin:admin@127.0.0.1:3000/api/dashboards/db -d @- \
          >/dev/null
    done

Related Reading

• Kernel Observability for the metric family catalog, label vocabulary, and log-line conventions every Chio crate emits.
• SIEM Export for receipt forwarding to Splunk HEC and Elasticsearch with dead-letter handling.
• TEE Deployment for the chio-tee Loki dashboard and TEE-specific signals.
• Receipt Dashboard for the receipt-first operator UI that is separate from Grafana.