Internet of Agents: Incident Network
Meridian Labs has a sev-1 outage on its inference gateway. A bad edge rule (geo-restrict-v42) on the CDN provider Stratos Networks is blocking legitimate traffic. A commander agent investigates, delegates a bounded fix across the org boundary, and the provider executes a single narrowed operation. Every tool call is mediated by chio. Every capability hop narrows the scope and the budget.
Where the code lives
examples/internet-of-agents-incident-network/. Run with ./smoke.sh. Set OPENAI_API_KEY or ANTHROPIC_API_KEY for a live agent loop, or leave both unset for the deterministic fallback used in CI.What It Shows
- Six chained capability hops: root commander, triage, change, vendor-liaison, provider coordinator, provider executor.
- Each hop is a delegation that narrows
grants,maxTotalCost,maxInvocations, andttl. - Cross-org dispatch through an ACP broker, with the provider coordinator forwarding into a bounded executor.
- Receipts and lineage chains for every capability, queryable through trust-control after the run.
- Five scenarios in one binary: happy path, attenuation deny, mid-chain revocation, approval required, capability TTL expiry.
Architecture
Service Inventory
smoke.sh picks free ports for everything. The shape:
| Process | Command | Role |
|---|---|---|
| trust-control | chio trust serve | Capability authority, revocation, receipt store, budget store. |
| mcp-observability | chio mcp serve-http + tools/observability.py | Read-only triage tools (incident summary, spans, deploy timeline, SLO). |
| mcp-github | chio mcp serve-http + tools/github.py | Search commits, get diff, get file. |
| mcp-pagerduty | chio mcp serve-http + tools/pagerduty.py | On-call state, escalation timeline. |
| mcp-provider-ops | chio mcp serve-http + tools/provider_ops.py | Disable edge rule (write tool, costed). |
| acp-broker | services/acp_broker.py | Cross-org task queue. |
| coordinator | services/coordinator.py | Provider entry point. Receives the ACP task and dispatches the executor. |
| executor | services/executor.py | Bounded operation runner. Validates the executor capability against trust-control. |
| coordinator sidecar | chio api protect | Receipt-signing front for the coordinator API. |
Run It
# From the chio workspace root
cargo build --bin chio
cd examples/internet-of-agents-incident-network
./smoke.sh
# Optional: live agent reasoning
export OPENAI_API_KEY=...
./smoke.sh
# Run a non-default scenario via orchestrate.py --mode
./scenario/01-happy-path.sh
./scenario/02-attenuation-deny.sh
./scenario/03-revoke-midchain.sh
./scenario/04-approval-required.sh
./scenario/05-expiry-async-failure.shPhase 1: Incident Trigger
The orchestrator loads a fixture incident from workspaces/customer-lab/incident/current-incident.json and persists it as the run header. It also generates an Ed25519 identity for every agent on the chain.
incident = json.loads(
(ROOT / "workspaces" / "customer-lab" / "incident" / "current-incident.json").read_text()
)
ids = {n: gen_identity(n) for n in [
"commander-agent", "triage-agent", "change-agent",
"vendor-liaison-agent", "provider-coordinator", "provider-executor",
]}Phase 2: Root Capability
Trust-control issues a root capability bound to the commander agent's public key. Read-only investigation tools are free. Cross-org engagement (acp-broker.create_task) carries a $10 budget at $5 per call.
root_cap = trust.issue_capability(
ids["commander-agent"].pk,
_scope(
_grant("mcp-observability", "get_incident_summary", ["invoke", "delegate"]),
_grant("mcp-observability", "query_spans", ["invoke", "delegate"]),
_grant("mcp-observability", "get_deploy_timeline", ["invoke", "delegate"]),
_grant("mcp-observability", "get_slo_status", ["invoke", "delegate"]),
_grant("mcp-github", "search_commits", ["invoke", "delegate"]),
_grant("mcp-github", "get_diff", ["invoke", "delegate"]),
_grant("mcp-github", "get_file", ["invoke", "delegate"]),
_grant("mcp-pagerduty", "get_oncall_state", ["invoke", "delegate"]),
_grant("mcp-pagerduty", "get_escalation_timeline", ["invoke", "delegate"]),
_grant("acp-broker", "create_task", ["invoke", "delegate"],
max_cost=1000, max_per_call=500),
),
ttl=1800,
)
trust.record_lineage(root_cap, None)Phase 3: Sub-Agent Delegation
The commander mints three narrowed delegations. The triage agent gets read-only investigation grants under ["invoke"] only (no further delegation). The change agent gets an empty scope (it reasons over the triage output, no tool calls). The vendor liaison gets a single grant: open one ACP task, $5 budget.
triage_cap = delegate(
parent=root_cap, delegator=ids["commander-agent"], delegatee=ids["triage-agent"],
scope=_scope(
_grant("mcp-observability", "get_incident_summary", ["invoke"]),
_grant("mcp-observability", "query_spans", ["invoke"]),
# ... investigation grants without "delegate" ...
),
ttl=900, cap_id=f"{incident['incident_id']}-triage",
)
vendor_cap = delegate(
parent=root_cap, delegator=ids["commander-agent"], delegatee=ids["vendor-liaison-agent"],
scope=_scope(_grant("acp-broker", "create_task", ["invoke", "delegate"],
max_cost=500, max_per_call=500)),
ttl=vendor_ttl, cap_id=f"{incident['incident_id']}-vendor",
)The narrowing is concrete. The parent root capability carries max_invocations: 100 on the cross-org grant and a 30-minute TTL. The vendor liaison cap narrows to a single invocation and 15 minutes; the coordinator cap narrows further to two invocations and 10 minutes. Concretely:
{
"id": "cap-019d921b-root-commander-...",
"subject": "f257309e...d6ed0",
"issuer": "f257309e...d6ed0",
"scope": {
"grants": [
{
"server_id": "acp-broker",
"tool_name": "create_task",
"operations": ["invoke", "delegate"],
"maxInvocations": 100,
"maxTotalCost": {"units": 1000, "currency": "USD"},
"maxCostPerInvocation": {"units": 500, "currency": "USD"}
}
]
},
"issued_at": 1776272775,
"expires_at": 1776274575,
"ttl_secs": 1800,
"delegation_depth": 0,
"parent_capability_id": null
}{
"id": "INC-2026-0428-vendor",
"subject": "vendor-liaison-pk-...",
"issuer": "f257309e...d6ed0",
"scope": {
"grants": [
{
"server_id": "acp-broker",
"tool_name": "create_task",
"operations": ["invoke", "delegate"],
"maxInvocations": 10,
"maxTotalCost": {"units": 500, "currency": "USD"},
"maxCostPerInvocation": {"units": 500, "currency": "USD"}
}
]
},
"issued_at": 1776272775,
"expires_at": 1776273675,
"ttl_secs": 900,
"delegation_depth": 1,
"parent_capability_id": "cap-019d921b-root-commander-..."
}maxInvocations dropped from 100 to 10. ttl_secs dropped from 1800 to 900. maxTotalCost dropped from $10 to $5. The kernel rejects any child that widens any of these dimensions; the trust-control issue_capability handler returns CHIO-CAP-NARROWING.
Phase 4: Investigation
The triage agent runs a tool-use loop against the chio MCP edges (observability, github, pagerduty). The change agent reasons over the triage output. The commander composes both into a decision. The vendor liaison decides whether to escalate to Stratos.
# Expected log lines (per MCP edge)
chio.mcp.serve-http: tools/call get_incident_summary -> allow
chio.mcp.serve-http: tools/call query_spans -> allow
chio.mcp.serve-http: receipt receipt_<id> sealedIf the live agent path is enabled, the model returns structured output naming the suspected rule. The deterministic fallback wires the same shape from the fixture data.
Phase 5: Cross-Org Dispatch
The orchestrator posts a task to the ACP broker. The broker hands it to the provider coordinator. The coordinator delegates a narrowed two-call capability to the executor and forwards the operation request:
coord_cap = delegate(
parent=vendor_cap, delegator=ids["vendor-liaison-agent"],
delegatee=PublicKey(name="provider-coordinator", pk=ids["provider-coordinator"].pk),
scope=_scope(_grant("provider-ops", "disable_edge_rule", ["invoke", "delegate"],
max_cost=500, max_per_call=500, max_invocations=2)),
ttl=600, cap_id=f"{task['task_id']}-provider-coordinator",
)At this point three capabilities sit on the chain in trust-control: commander <- vendor-liaison <- coordinator <- executor. Each step narrows. trust.record_lineage(child, parent_id) makes the chain queryable.
The vendor-liaison agent sends an ACP task to the broker. The broker forwards a coordinator-shaped request into the provider coordinator endpoint:
{
"task_id": "task_INC-2026-0428_001",
"incident_id": "INC-2026-0428",
"target_service": "inference-gateway",
"target_rule": "geo-restrict-v42",
"bounded_action": "disable_rule",
"provider_instructions": "Disable the geo-restrict rule for the affected region.",
"vendor_liaison_capability": { "id": "INC-2026-0428-vendor", "...": "..." },
"execution_deadline": 1776273675
}{
"task": {
"task_id": "task_INC-2026-0428_001",
"target_service": "inference-gateway",
"target_rule": "geo-restrict-v42",
"bounded_action": "disable_rule",
"provider_coordinator_capability": {
"id": "task_INC-2026-0428_001-provider-coordinator",
"delegation_depth": 2,
"parent_capability_id": "INC-2026-0428-vendor"
}
},
"provider_coordinator_seed_hex": "...",
"provider_executor_public_key": "executor-pk-...",
"control_url": "http://127.0.0.1:<TRUST_PORT>",
"service_token": "demo-token",
"provider_executor_url": "http://127.0.0.1:<EXEC_PORT>",
"executor_ttl_seconds": 600
}Phase 6: Bounded Execution
The executor calls disable_edge_rule on mcp-provider-ops, presents its capability to chio, and the edge admits the call. The provider workspace state at workspaces/provider-lab/tenants/MeridianLabs/services/edge-global.json flips to reflect the disabled rule. An audit-log entry lands at workspaces/provider-lab/operations/audit-log.json.
The orchestrator marks the ACP task complete and pulls back the finalized record.
Phase 7: Audit
After execution, the orchestrator pulls back the lineage chain for every capability and writes financial reports:
for label, cid in cap_ids.items():
_write(out / "lineage" / f"{label}-chain.json", trust.lineage_chain(cid))
budget_state = trust.query_budgets()
_write(out / "financial" / "budget-usage.json", budget_state)
exposure = trust.exposure_ledger(agent_subject=executor_pk)
_write(out / "financial" / "exposure-ledger.json", exposure)
scorecard = trust.credit_scorecard(agent_subject=executor_pk)
_write(out / "financial" / "credit-scorecard.json", scorecard)
settlements = trust.settlement_report()
_write(out / "financial" / "settlement-report.json", settlements)Scenarios
The same orchestrator runs five scenarios under --mode:
| Mode | What it tests | Driver |
|---|---|---|
happy-path | Full six-hop delegation lands the fix. | scenario/01-happy-path.sh |
attenuation-deny | Executor asks for a broader rule than the chain allows. Edge denies. | scenario/02-attenuation-deny.sh |
revoke-midchain | Upstream cap is revoked while the executor is mid-call. Revocation propagates. | scenario/03-revoke-midchain.sh |
approval-required | Broader rollback needs a signed approval before dispatch. | scenario/04-approval-required.sh |
expiry-async-failure | Vendor cap TTL is set to 2s. The chain expires before the executor runs. | scenario/05-expiry-async-failure.sh |
Capability Lineage Query
After the run, every capability id has a six-hop chain. Pull the chain with the chio CLI:
chio --control-url "$CONTROL_URL" --control-token "$SERVICE_TOKEN" \
capability lineage \
--root cap-019d921b-root-commander-... \
--json | jq '.chain | map({id, delegation_depth, parent_capability_id})'[
{ "id": "cap-019d921b-root-commander-...", "delegation_depth": 0,
"parent_capability_id": null },
{ "id": "INC-2026-0428-triage", "delegation_depth": 1,
"parent_capability_id": "cap-019d921b-root-commander-..." },
{ "id": "INC-2026-0428-change", "delegation_depth": 1,
"parent_capability_id": "cap-019d921b-root-commander-..." },
{ "id": "INC-2026-0428-vendor", "delegation_depth": 1,
"parent_capability_id": "cap-019d921b-root-commander-..." },
{ "id": "task_INC-2026-0428_001-provider-coordinator", "delegation_depth": 2,
"parent_capability_id": "INC-2026-0428-vendor" },
{ "id": "task_INC-2026-0428_001-provider-executor", "delegation_depth": 3,
"parent_capability_id": "task_INC-2026-0428_001-provider-coordinator" }
]Mid-Chain Revocation
The revoke-midchain scenario revokes the vendor-liaison cap after the executor cap has been issued but before the executor calls disable_edge_rule. The executor presents its own valid cap, but the kernel walks the parent chain and finds a revoked ancestor:
{
"capability_id": "INC-2026-0428-vendor",
"revoked": true,
"newly_revoked": true,
"revoked_at": 1776272790
}{
"execution": {
"verdict": "deny",
"reason": "ancestor capability revoked",
"revoked_capability_id": "INC-2026-0428-vendor",
"denied_at_depth": 1
}
}The downstream call is denied. The kernel writes a denial receipt identifying which ancestor was revoked and at what delegation depth. No silent allow.
Smoke Assertions
Like the commerce example, the smoke runs the in-tree verifier after the orchestrator. The verifier reads the bundle manifest, confirms every required artifact exists, walks the lineage chain for each capability, and checks that the executor receipt's cost matches the consumed budget:
uv run --project "${EXAMPLE_ROOT}" python -c "
import sys; sys.path.insert(0, '${EXAMPLE_ROOT}')
from incident_network.verify import verify_bundle
import json
r = verify_bundle('${ARTIFACT_ROOT}')
json.dump(r, open('${ARTIFACT_ROOT}/review-result.json', 'w'), indent=2)
assert r['ok'], r['errors']
"
printf 'internet-of-agents-incident-network smoke passed\n'
printf 'artifacts: %s\n' "${ARTIFACT_ROOT}"Inspect After
cd artifacts/live/<timestamp>
export TRUST_DB="$(pwd)/state/trust-receipts.sqlite3"
export REVOKE_DB="$(pwd)/state/trust-revocations.sqlite3"
export BUDGET_DB="$(pwd)/state/trust-budgets.sqlite3"
# Receipts grouped by tool, in invocation order
sqlite3 "$TRUST_DB" \
'select tool_server, tool_name, verdict, count(*) \
from receipts group by tool_server, tool_name;'
# Revocation table after a revoke-midchain run
sqlite3 "$REVOKE_DB" \
'select capability_id, revoked_at from revocations;'
# Budget consumption per grant
sqlite3 "$BUDGET_DB" \
'select capability_id, total_charged, total_settled from budget_state;'
# Lineage projection (one file per capability)
ls lineage/
cat lineage/provider-executor-chain.json | jq '.chain | length'
# 4 -- root, vendor, coordinator, executor
# Final ACP task state
cat acp/task-final.json | jq '{status, completed_at}'Federation Note
The default smoke runs every chio component in one process tree and one trust-control. The cross-org boundary is enforced by the ACP broker and the chained capability scope, not by separate trust domains. To run this example with two real trust-control instances, see Bootstrap Federated Trust for the handshake and Bilateral Federation for the receipt-pair semantics across federated kernels.
Inspect the Bundle
cd artifacts/live/<timestamp>
cat incident.json
cat identities/public-identities.json
# One file per capability on the chain
ls capabilities/
cat capabilities/root-commander.json
cat capabilities/triage-agent.json
cat capabilities/provider-executor.json
# One lineage chain per capability
ls lineage/
cat lineage/provider-executor-chain.json
# Agent decisions
cat agents/triage-output.json
cat agents/commander-output.json
cat agents/vendor-liaison-output.json
# ACP task lifecycle
cat acp/task-created.json
cat acp/task-final.json
cat provider/process-task-response.json
# Financial reports from trust-control
cat financial/budget-usage.json
cat financial/exposure-ledger.json
cat financial/credit-scorecard.json
cat financial/settlement-report.json
# Manifest hash list for offline verification
cat bundle-manifest.json
cat summary.jsonThe smoke runs incident_network.verify.verify_bundle at the end. It rehashes every required artifact, walks the lineage chains, and asserts that the cost on the executor receipt matches the consumed budget. Output lands in review-result.json.
Use this example when...
Where to read more