OpenTelemetry

Chio receipts and OTel spans share two anchors: chio.receipt.id on the span and provenance.otel.span_id on the receipt. The otel-genai example starts a collector / Tempo / Jaeger / Grafana stack, runs an ignored Rust integration test that exports a fully-populated GenAI tool-call span through chio-otel-receipt-exporter, and proves both lookup directions work.

Where the code lives

examples/otel-genai/. The collector demo uses docker compose up; the contract test runs under cargo test ... -- --ignored.

What It Shows

OTel collector wired to Tempo (TraceQL) and Jaeger (lookup by trace ID and tag) with Grafana on top.
The locked GenAI tool-call attribute set chio targets: gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.tool.call.id, gen_ai.tool.name, plus chio attributes (chio.receipt.id, chio.tenant.id, chio.policy.ref, chio.verdict, chio.tee.mode) and OTel provenance (provenance.otel.trace_id, provenance.otel.span_id).
Bidirectional lookup: receipt id resolves to span id and back, end to end through the exporter.
Metric-safety policy on the collector: high-cardinality attributes are stripped from metric pipelines while the receipt-id field on traces is preserved.

Prerequisites

Docker with docker compose v2.
Rust toolchain (only for the contract test).
Optional: jq and curl for the dashboard import script.

Files

text

examples/otel-genai/
  Cargo.toml
  README.md
  docker-compose.yml
  otel-collector-config.yaml
  tests/bidirectional_lookup.rs

Run the Collector Demo

bash

cd examples/otel-genai
docker compose up

The compose file boots four services. Pinned image versions in the repo:

examples/otel-genai/docker-compose.yml

services:
  otel-collector:                  # otel/opentelemetry-collector-contrib:0.115.1
    ports: ["4317:4317", "4318:4318", "8889:8889"]
    depends_on: [tempo, jaeger]

  tempo:                           # grafana/tempo:2.6.1
    ports: ["3200:3200"]           # OTLP grpc/http on 4317/4318 inside the network

  jaeger:                          # jaegertracing/all-in-one:1.62
    ports: ["16686:16686"]
    environment: { COLLECTOR_OTLP_ENABLED: "true" }

  grafana:                         # grafana/grafana:11.5.0
    ports: ["3000:3000"]
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    depends_on: [tempo, jaeger]

The compose stack exposes:

Service	Endpoint	Purpose
OTLP gRPC	`127.0.0.1:4317`	GenAI spans from adapters or local clients.
OTLP HTTP	`127.0.0.1:4318`	HTTP OTLP ingest for local tooling.
Jaeger UI	`http://127.0.0.1:16686`	Lookup by trace ID, receipt-id tag, span-id tag.
Tempo	`http://127.0.0.1:3200`	TraceQL by `span.chio.receipt.id` and `span.chio.verdict`.
Grafana	`http://127.0.0.1:3000`	Dashboard host. Anonymous admin role; password `admin`.

Import the chio dashboards from the repository root:

bash

find deploy/dashboards -name '*.json' -print0 \
  | while IFS= read -r -d '' dashboard; do
      jq -n --argjson dashboard "$(cat "$dashboard")" \
        '{dashboard: $dashboard, overwrite: true}' \
      | curl -fsS -H 'Content-Type: application/json' \
          -X POST http://admin:admin@127.0.0.1:3000/api/dashboards/db -d @- >/dev/null
    done

Collector Pipeline

The collector receives OTLP on gRPC and HTTP, fans traces out to Tempo, Jaeger, and a debug exporter, and uses an attributes processor to strip high-cardinality keys from the metric pipeline. The receipt-id stays on the trace pipeline; the metric pipeline drops it so a Prometheus-shaped backend does not see it as a label:

examples/otel-genai/otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}
  attributes/chio-metric-safety:
    actions:
      - key: gen_ai.tool.call.id
        action: delete
      - key: chio.receipt.id
        action: delete
      - key: chio.replay.run_id
        action: delete

exporters:
  debug:
    verbosity: basic
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [attributes/chio-metric-safety, batch]
      exporters: [debug]

Why drop receipt id from metrics

Receipt id is unique per evaluation. Letting it land as a metric label produces unbounded cardinality in any Prometheus-shaped backend. The collector keeps it on traces (where unique IDs are the point) and strips it from the metric pipeline.

Run the Contract Test

From the chio workspace root:

bash

cargo test --manifest-path examples/otel-genai/Cargo.toml \
  --test bidirectional_lookup -- --ignored

The test constructs a decoded OTLP trace export with the locked GenAI tool-call attributes, exports it through chio-otel-receipt-exporter, verifies the signed receipt, and builds both lookup directions:

receipt id -> span id (read provenance.otel.span_id out of the receipt metadata)
span id -> receipt id (read chio.receipt.id attribute off the span)

It also confirms the metric-safety policy: high-cardinality attributes that the collector strips from metrics never appear in the receipt metadata copy, while chio.receipt.id remains the signed receipt identifier.

Expected Span Shape

The contract test sends a single span called gen_ai.tool.call with these attributes (verbatim from the test source):

examples/otel-genai/tests/bidirectional_lookup.rs

let span = OtlpSpan::new(trace_id, span_id, "gen_ai.tool.call")
    .with_attribute("gen_ai.system", serde_json::json!("openai"))
    .with_attribute("gen_ai.operation.name", serde_json::json!("tool.call"))
    .with_attribute("gen_ai.request.model", serde_json::json!("gpt-5"))
    .with_attribute("gen_ai.tool.call.id", serde_json::json!("call-demo-1"))
    .with_attribute("gen_ai.tool.name", serde_json::json!("customer_lookup"))
    .with_attribute("gen_ai.usage.input_tokens", serde_json::json!(42))
    .with_attribute("gen_ai.usage.output_tokens", serde_json::json!(7))
    .with_attribute("chio.receipt.id", serde_json::json!(receipt_id))
    .with_attribute("chio.tenant.id", serde_json::json!("tenant-demo"))
    .with_attribute("chio.policy.ref", serde_json::json!("policy-demo-otel"))
    .with_attribute("chio.verdict", serde_json::json!("allow"))
    .with_attribute("chio.tee.mode", serde_json::json!("shadow"))
    .with_attribute("chio.capability.id", serde_json::json!("cap-otel-demo"))
    .with_attribute("chio.server.id", serde_json::json!("srv-openai-demo"))
    .with_attribute("chio.agent.id", serde_json::json!("agent-demo"));

The same attribute set is what the kernel's OTel helpers emit. Adapters that decorate spans with these attributes are compatible with the collector pipeline and the dashboards out of the box.

Sample Queries

With the collector running and traces flowing, slice the data from Tempo or Jaeger:

text

# TraceQL (Tempo): all denied tool calls in the last hour
{ span.chio.verdict = "deny" }

# TraceQL: lookup by receipt id
{ span.chio.receipt.id = "rcpt-otel-genai-0001" }

# TraceQL: deny rate by policy
{ span.chio.verdict = "deny" } | by(span.chio.policy.ref)

# TraceQL: latency tail by tool
{ span.gen_ai.operation.name = "tool.call" } | quantile_over_time(span.duration, 0.99) by(span.gen_ai.tool.name)

# Jaeger: search by tag
chio.receipt.id="rcpt-otel-genai-0001"
chio.verdict="deny"

Fuel and budget metrics flow on the metric pipeline (with receipt-id stripped). The chio dashboards under deploy/dashboards/ chart deny rate, fuel consumption per policy, and the p50/p99 latency tail per server.

On the metric side, the kernel histogram chio_guard_eval_duration_seconds is the most common Prometheus query. p99 by guard:

prometheus query

histogram_quantile(
  0.99,
  sum by (le, guard_name)(
    rate(chio_guard_eval_duration_seconds_bucket[5m])
  )
)

# Sample response (one series per guard):
# {guard_name="hushspec_tool_access"}     0.00042
# {guard_name="hushspec_argument_filter"} 0.00018
# {guard_name="capability_check"}         0.00005

Wire It Into Your Service

To produce the same shape from a service running outside the kernel, pull chio-otel-receipt-exporter into your binary and emit spans with the attribute set above. The collector deduplicates by trace + span id, so the receipt sink always lines up with the exported span.

Decision rule

Use this stack when you want a fully working trace + metric path that already strips high-cardinality keys before they reach the metric pipeline. Pick SIEM Export instead when the goal is durable evidence storage in Splunk or Elastic; SIEM is the long-tail audit channel, OTel is the live operational signal. Don't drop the metric-safety attributes processor in production: receipt id is unique per evaluation and will blow up label cardinality on any Prometheus-shaped backend.

Where to read more

Observability for the full kernel telemetry surface. SIEM Export for moving receipts into Splunk and Elastic on a separate cadence; the OTel path is the live operational signal, the SIEM path is the long-tail evidence stream.

PreviousReceipt Verification NextIstio ext_authz