Observability

Tracing & Structured Logging

Every chio crate uses the tracing crate. Guards, kernel evaluation, and adapters emit structured events through five standard macros:

• tracing::trace! · per-step state in evaluation. Off in production.
• tracing::debug! · per-call detail useful for local development. Off in production unless investigating.
• tracing::info! · lifecycle: kernel started, policy reloaded, checkpoint written, revocation propagated.
• tracing::warn! · recoverable degradation and fail-closed fallbacks: cache-miss escalation, a guard rejecting invalid config and constructing a deny-all fallback (e.g. SqlQueryGuard::new on an over-broad denylist pattern), external-guard call failures, advisory verdicts.
• tracing::error! · fail-closed events that produce Verdict::Deny via the fallback path: caught panics, poisoned mutexes, signing failures, store unreachable.

There is no single fixed field set across guards. The guards that emit tracing share one convention — a guard field carrying the guard's kebab-case Guard::name() — and otherwise attach whatever fields describe that guard's decision. Many deterministic guards, ForbiddenPathGuard among them, emit no tracing at all; their evidence lives in the receipt, not the log. Two representative call sites:

tracing::warn!(
    guard = "code-execution",
    module = %name,
    "denying code execution: dangerous module detected",
);

tracing::warn!(
    guard = "content-review",
    service = %service,
    endpoint = %endpoint,
    detected_categories = ?categories,
    "content-review denied outbound message",
);

The guard field is the one stable join key. Its value is the same kebab-case string the guard reports through Guard::name() ("code-execution", "content-review", "forbidden-path") and the same string the pipeline records as GuardEvidence.guard_name on the receipt, so a log line and its receipt correlate on that value. The remaining fields (module, service, endpoint, code_len, ...) are guard-specific; do not assume a decision, path, or reason field on an arbitrary guard's events.

WASM Guard Metrics

Custom WASM guards emit a Prometheus-shaped metric family registered through GuardMetricRegistry. The family descriptors are static constants in chio_wasm_guards::metrics:

Histogram Buckets

Eval-duration buckets are tuned for WASM guard latencies:

pub const EVAL_DURATION_BUCKETS_SECONDS: &[f64] = &[
    0.0001, 0.0005, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0,
];

pub const HOST_CALL_DURATION_BUCKETS_SECONDS: &[f64] = &[
    0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1,
];

Override these only if your dashboards demand different SLO breakpoints. They are shared by every guard so cross-guard p99 comparisons are meaningful out of the box.

Label Vocabulary

The label values are closed sets, bounded so each series has a finite label domain:

// chio_guard_verdict_total.verdict
pub const VERDICT_LABEL_VALUES: &[&str] =
    &[VERDICT_ALLOW, VERDICT_DENY, VERDICT_REWRITE, VERDICT_ERROR];

// chio_guard_deny_total.reason_class
pub const REASON_CLASS_LABEL_VALUES: &[&str] = &[
    "policy", "pii", "secret", "prompt_injection",
    "oversize", "fuel", "trap", "malformed", "other",
];

// chio_guard_host_call_duration_seconds.host_fn
pub const HOST_FN_LABEL_VALUES: &[&str] = &[
    HOST_LOG, HOST_GET_CONFIG, HOST_GET_TIME_UNIX_SECS, HOST_FETCH_BLOB,
];

// chio_guard_reload_total.outcome
pub const RELOAD_OUTCOME_LABEL_VALUES: &[&str] =
    &[RELOAD_APPLIED, RELOAD_CANARY_FAILED, RELOAD_ROLLED_BACK];

Two of the nine reason_class values are the bounding valves, not error states. malformed marks a deny where host-side argument extraction failed before the guard module ran. other is the deliberate fallback that keeps chio_guard_deny_total{reason_class} finite even when a guard emits a novel free-form reason: classify_deny_reason_class maps any unrecognized or absent reason to it. Seeing reason_class="other" on a dashboard means the classifier assigned the fallback category.

Cardinality Limits

The registry caps unique guard IDs at MAX_GUARD_METRIC_CARDINALITY = 1024. Beyond that, registration returns GuardMetricRegistrationError with code E_GUARD_METRIC_CARDINALITY_EXCEEDED and the guard is dropped from the metric family.

The registry derives a stable 12-character guard ID from the guard digest:

pub fn guard_id_label_from_digest(digest: &str) -> String {
    digest
        .strip_prefix("sha256:")
        .unwrap_or(digest)
        .chars()
        .take(12)
        .collect()
}

Two guards with the same first 12 hex digits would collide. In practice this is a non-issue at 1024 active guards; if your fleet approaches the cap, raise the cardinality limit explicitly via GuardMetricRegistry::with_max_guards(n) instead of truncating it to fewer characters.

OpenTelemetry Receipt Exporter

The crate chio-otel-receipt-exporter accepts OTLP trace batches and writes derived chio receipts to a configured store. Two pieces:

• OtlpGrpcIngress · accepts OTLP/gRPC trace exports in a narrow Rust representation.
• ReceiptStoreSink · builds a ChioReceipt per OTLP span, signs it via ReceiptStoreSinkConfig.signing_keypair, and appends it to the configured Arc<dyn ReceiptStore>.

pub struct ReceiptStoreSinkConfig {
    pub signing_keypair: Keypair,
    // ... additional kernel-key, tenant, schema fields
}

pub struct ReceiptStoreSink { /* ... */ }

impl ReceiptStoreSink {
    pub fn new(store: Arc<dyn ReceiptStore>, config: ReceiptStoreSinkConfig) -> Self;
    pub fn export_traces(&self, export: &OtlpGrpcTraceExport)
        -> Result<ReceiptStoreSinkSummary, OTelReceiptExportError>;
    pub fn receipt_for_span(&self, span: &OtlpSpan)
        -> Result<ChioReceipt, OTelReceiptExportError>;
}

High-Cardinality Attribute Denylist

Span attributes that would explode Prometheus cardinality (request IDs, user IDs, raw URLs) are stripped before forwarding to Prometheus-shaped sinks. The denylist is exposed via:

pub use denylist::{
    denied_attribute_keys,
    is_denied_attribute,
    strip_denied_attributes,
    strip_denied_batch_attributes,
    strip_denied_span_attributes,
    PROMETHEUS_DENIED_ATTRIBUTES,
};

Use strip_denied_span_attributes before pushing spans to a Prometheus exporter; the denylist is opinionated about the keys that produce unbounded series.

Signal Hierarchy

Three signals, three roles:

For dispute resolution and compliance evidence, use receipts. For "is the kernel healthy right now?", use metrics. For "why did request X fail at 03:14", use tracing joined to the receipt for X by capability ID.

Operational Dashboards

Four signals matter on day-to-day dashboards:

• Deny rate per guard. rate(chio_guard_deny_total[5m]) grouped by guard_id, reason_class. Alert when a guard's deny rate jumps above its baseline; this is the early-warning channel for both attack signals and policy regressions.
• Fuel consumption tail. histogram_quantile(0.99, ...) on chio_guard_eval_duration_seconds with verdict="deny", reason_class="fuel". Indicates WASM modules approaching their fuel ceiling.
• External-guard call failures. The breaker state machine itself does not log transitions. What you alert on is the adapter's failure path: every failed provider call logs tracing::warn!(guard, error, "external guard failed") and trips the circuit breaker toward open. Build an alert on the rate of these per guard; a sustained rate means a provider is degraded and the breaker is shedding load.
• Reload outcomes. chio_guard_reload_total grouped by outcome. A rising canary_failed or rolled_back count means hot-reload deployments are not landing cleanly.

Next Steps

• Receipts & Audit · the canonical audit signal that metrics and traces complement
• Performance & Tuning · which signals point at which bottlenecks
• Failure & Recovery · how to read circuit-breaker telemetry under load
• Custom Guards · how a guard registers its own metrics