OpenTelemetry Example

Where the code lives

examples/otel-genai/. The collector demo uses docker compose up; the contract test runs under cargo test ... -- --ignored.

What It Shows

OTel collector wired to Tempo (TraceQL) and Jaeger (lookup by trace ID and tag) with Grafana on top.
The locked GenAI tool-call attribute set chio targets: gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.tool.call.id, gen_ai.tool.name, plus chio attributes (chio.receipt.id, chio.tenant.id, chio.policy.ref, chio.verdict, chio.tee.mode) and OTel provenance (provenance.otel.trace_id, provenance.otel.span_id).
Bidirectional lookup between the receipt ID and span ID through the exporter.
Metric-safety policy on the collector: high-cardinality attributes are stripped from metric pipelines while the receipt-id field on traces is preserved.

Prerequisites

Docker with docker compose v2.
Rust toolchain (only for the contract test).
Optional: jq and curl for the dashboard import script.

Files

text

examples/otel-genai/
  Cargo.toml
  README.md
  docker-compose.yml
  otel-collector-config.yaml
  tests/bidirectional_lookup.rs

Run the Collector Demo

bash

cd examples/otel-genai
docker compose up

The compose file boots four services. Pinned image versions in the repo:

examples/otel-genai/docker-compose.yml

services:
  otel-collector:                  # otel/opentelemetry-collector-contrib:0.115.1
    ports: ["4317:4317", "4318:4318", "8889:8889"]
    depends_on: [tempo, jaeger]

  tempo:                           # grafana/tempo:2.6.1
    ports: ["3200:3200"]           # OTLP grpc/http on 4317/4318 inside the network

  jaeger:                          # jaegertracing/all-in-one:1.62
    ports: ["16686:16686"]
    environment: { COLLECTOR_OTLP_ENABLED: "true" }

  grafana:                         # grafana/grafana:11.5.0
    ports: ["3000:3000"]
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
      GF_SECURITY_ADMIN_PASSWORD: admin
    depends_on: [tempo, jaeger]

The compose stack exposes:

Service	Endpoint	Purpose
OTLP gRPC	`127.0.0.1:4317`	GenAI spans from adapters or local clients.
OTLP HTTP	`127.0.0.1:4318`	HTTP OTLP ingest for local tooling.
Jaeger UI	`http://127.0.0.1:16686`	Lookup by trace ID, receipt-id tag, span-id tag.
Tempo	`http://127.0.0.1:3200`	TraceQL by `span.chio.receipt.id` and `span.chio.verdict`.
Grafana	`http://127.0.0.1:3000`	Dashboard host. Anonymous admin role; password `admin`.

Import the Chio dashboards from the repository root:

bash

find deploy/dashboards -name '*.json' -print0 \
  | while IFS= read -r -d '' dashboard; do
      jq -n --argjson dashboard "$(cat "$dashboard")" \
        '{dashboard: $dashboard, overwrite: true}' \
      | curl -fsS -H 'Content-Type: application/json' \
          -X POST http://admin:admin@127.0.0.1:3000/api/dashboards/db -d @- >/dev/null
    done

Collector Pipeline

The collector receives OTLP on gRPC and HTTP, fans traces out to Tempo, Jaeger, and a debug exporter, and uses an attributes processor to strip high-cardinality keys from the metric pipeline. The receipt-id stays on the trace pipeline; the metric pipeline drops it so a Prometheus-shaped backend does not see it as a label:

examples/otel-genai/otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}
  attributes/chio-metric-safety:
    actions:
      - key: gen_ai.tool.call.id
        action: delete
      - key: chio.receipt.id
        action: delete
      - key: chio.replay.run_id
        action: delete

exporters:
  debug:
    verbosity: basic
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo, otlp/jaeger, debug]
    metrics:
      receivers: [otlp]
      processors: [attributes/chio-metric-safety, batch]
      exporters: [debug]

Why drop receipt id from metrics

Receipt id is unique per evaluation. Letting it land as a metric label produces unbounded cardinality in a Prometheus-compatible backend. The collector keeps it on traces and strips it from metrics.

Run the Contract Test

From the Chio workspace root:

bash

cargo test --manifest-path examples/otel-genai/Cargo.toml \
  --test bidirectional_lookup -- --ignored

The test constructs a decoded OTLP trace export with the locked GenAI tool-call attributes, exports it through chio-otel-receipt-exporter, verifies the signed receipt, and builds both lookup directions:

receipt id -> span id (read provenance.otel.span_id out of the receipt metadata)
span id -> receipt id (read chio.receipt.id attribute off the span)

It also confirms the metric-safety policy: high-cardinality attributes that the collector strips from metrics never appear in the receipt metadata copy, while chio.receipt.id remains the signed receipt identifier.

Expected Span Shape

The contract's demo_span() helper (called from the test through export_demo_span()) builds a single span called gen_ai.tool.call with these attributes:

examples/otel-genai/tests/support/mod.rs (demo_span)

let span = OtlpSpan::new(trace_id, span_id, "gen_ai.tool.call")
    .with_attribute("gen_ai.system", serde_json::json!("openai"))
    .with_attribute("gen_ai.operation.name", serde_json::json!("tool.call"))
    .with_attribute("gen_ai.request.model", serde_json::json!("gpt-5"))
    .with_attribute("gen_ai.tool.call.id", serde_json::json!("call-demo-1"))
    .with_attribute("gen_ai.tool.name", serde_json::json!("customer_lookup"))
    .with_attribute("gen_ai.usage.input_tokens", serde_json::json!(42))
    .with_attribute("gen_ai.usage.output_tokens", serde_json::json!(7))
    .with_attribute("chio.receipt.id", serde_json::json!(receipt_id))
    .with_attribute("chio.tenant.id", serde_json::json!("tenant-demo"))
    .with_attribute("chio.policy.ref", serde_json::json!("policy-demo-otel"))
    .with_attribute("chio.verdict", serde_json::json!("allow"))
    .with_attribute("chio.tee.mode", serde_json::json!("shadow"))
    .with_attribute("chio.capability.id", serde_json::json!("cap-otel-demo"))
    .with_attribute("chio.server.id", serde_json::json!("srv-openai-demo"))
    .with_attribute("chio.agent.id", serde_json::json!("agent-demo"))
    .with_attribute("redaction_pass_id", serde_json::json!("redactors@1.5.0+default"))
    .with_attribute("redaction_elapsed_micros", serde_json::json!(12450_u64));

The same attribute set is what the kernel's OTel helpers emit. The two redaction_* attributes record the redaction pass that ran before the receipt was signed — redaction_pass_id pins the redactor set and version, redaction_elapsed_micros its wall-clock cost. Adapters that decorate spans with these attributes are compatible with the collector pipeline and the dashboards out of the box.

Sample Queries

With the collector running and traces flowing, slice the data from Tempo or Jaeger:

text

# TraceQL (Tempo): all denied tool calls in the last hour
{ span.chio.verdict = "deny" }

# TraceQL: lookup by receipt id
{ span.chio.receipt.id = "rcpt-otel-genai-0001" }

# TraceQL: deny rate by policy
{ span.chio.verdict = "deny" } | by(span.chio.policy.ref)

# TraceQL: latency tail by tool
{ span.gen_ai.operation.name = "tool.call" } | quantile_over_time(span.duration, 0.99) by(span.gen_ai.tool.name)

# Jaeger: search by tag
chio.receipt.id="rcpt-otel-genai-0001"
chio.verdict="deny"

Fuel and budget metrics flow on the metric pipeline (with receipt-id stripped). The chio dashboards under deploy/dashboards/ chart deny rate, fuel consumption per policy, and the p50/p99 latency tail per server.

On the metric side, the kernel histogram chio_guard_eval_duration_seconds is the most common Prometheus query. Its locked label set is guard_id and verdict. p99 by guard and verdict:

prometheus query

histogram_quantile(
  0.99,
  sum by (le, guard_id, verdict)(
    rate(chio_guard_eval_duration_seconds_bucket[5m])
  )
)

# Sample response (label set is guard_id, verdict):
# {guard_id="tool-gate",          verdict="allow"} 0.00042
# {guard_id="tool-gate",          verdict="deny"}  0.00061
# {guard_id="enriched-inspector", verdict="allow"} 0.00018

Wire It Into Your Service

To produce the same shape from a service running outside the kernel, pull chio-otel-receipt-exporter into your binary and emit spans with the attribute set above. The collector deduplicates by trace + span id, so the receipt sink can match the exported span.

Decision rule

Use this stack for traces and metrics with high-cardinality keys removed before metric export. Use SIEM Export for durable evidence storage in Splunk or Elastic. Keep the metric-safety attributes processor in production: a receipt ID is unique per evaluation and would create unbounded label cardinality.

Where to read more

Observability for kernel telemetry. SIEM Export for moving receipts into Splunk and Elastic.

PreviousReceipt Verification NextIstio ext_authz