Failure & Recovery

Failure-Mode Matrix

Reason classification is narrower than this matrix. Only WASM guard denials carry a bounded reason_class label, on the chio_guard_deny_total{guard_id, reason_class} counter: chio-wasm-guards is the sole producer; the kernel /metrics endpoint only renders it. The label domain is fixed to nine values: policy, pii, secret, prompt_injection, oversize, fuel, trap, malformed, other, assigned by substring-matching the guard's free-form deny reason, with any unrecognized reason folded into other. Native-guard, circuit-breaker, mutex-poisoning, session-journal, and receipt-store denials in the table above carry no reason_class.

Circuit Breaker

External-guard adapters wrap their inner provider call in a three-state breaker (crates/guards/chio-guards/src/external/circuit_breaker.rs):

pub enum CircuitState {
    Closed,    // Normal operation. Failures count toward a sliding window.
    Open,      // Fail-fast. Calls short-circuit until reset_timeout elapses.
    HalfOpen,  // Probing. A bounded number of trial calls test recovery.
}

The transitions:

• Closed to Open. When the number of failures inside the rolling failure_window reaches failure_threshold.
• Open to HalfOpen. When reset_timeout has elapsed since the breaker opened. The next call admitted moves to HalfOpen.
• HalfOpen to Closed. After success_threshold consecutive successes.
• HalfOpen to Open. Any single failure during probing reopens the breaker.

pub struct CircuitBreakerConfig {
    pub failure_threshold: u32,    // default 5
    pub failure_window: Duration,  // default 60s
    pub success_threshold: u32,    // default 2
    pub reset_timeout: Duration,   // default 30s
}

CircuitOpenVerdict

When the breaker is Open, the adapter does not call the provider. Instead it returns a configurable verdict:

pub enum CircuitOpenVerdict {
    Deny,   // Fail-closed. Default.
    Allow,  // Fail-open. Advisory only.
}

Use Allow when unavailability is preferable to a denied request. Two appropriate cases are:

• The guard's output feeds a human-review queue rather than gating an action. A missed signal is recoverable.
• The guard is one of several independent layers and the others are still in the synchronous chain. The remaining synchronous guards continue to enforce the action.

RateLimitedVerdict

Same shape, different trigger. When the adapter's token bucket is empty:

pub enum RateLimitedVerdict {
    Deny,   // Default. The QPS budget is exhausted; deny the request.
    Allow,  // Advisory: allow the request even though the budget is exhausted.
}

Retry Strategies

Adapters retry transient and timeout errors only. 4xx and malformed-request errors are permanent; they short-circuit the retry loop and become Verdict::Deny.

pub enum BackoffStrategy {
    Exponential,  // base_delay * 2^(attempt - 1)
    Constant,     // base_delay
    Linear,       // base_delay * attempt
}

pub struct RetryConfig {
    pub max_retries: u32,         // default 3 (so 4 total attempts)
    pub base_delay: Duration,     // default 100ms
    pub max_delay: Duration,      // default 5s
    pub jitter_fraction: f64,     // default 0.25 (clamped to [0.0, 1.0])
    pub strategy: BackoffStrategy,// default Exponential
}

Three things to know:

• Total attempts is max_retries + 1. A value of 0 means "run exactly once, no retries".
• Jitter is bounded multiplicative. The configured delay is multiplied by 1 + uniform(-jitter_fraction, +jitter_fraction). Defaults give a ±25% spread, which is enough to avoid thundering herd while keeping the worst-case bounded.
• Jitter is deterministic by default. The retry RNG is seeded from max_retries so tests reproduce. Production callers wanting non-deterministic jitter can use retry_with_jitter_rng.

Composed Adapter Flow

Inside AsyncGuardAdapter, the failure-handling pieces compose in a fixed order:

evaluate(ctx):
    1. CircuitBreaker.allow_call()          -> CircuitOpenVerdict on deny
    2. TtlCache.get(cache_key)              -> cached verdict on hit
    3. TokenBucket.try_acquire()            -> RateLimitedVerdict on empty
    4. retry_with_jitter(inner.eval)        -> Verdict::Deny on permanent failure
                                             -> Verdict on success (also cached)

Two invariants matter:

• Cache hits do not spend rate-limit budget. The cache check precedes the token bucket on purpose: a steady-state hot cache is free.
• Rate-limited calls do not count as breaker failures. Only attempted calls to the external service feed the failure window. A bursty client cannot trip the breaker by exceeding the rate limit.

Recovery Patterns

Hot-Reload a WASM Guard

A WASM guard that hits its fuel ceiling or traps repeatedly is quarantined: the bundle store marks it unhealthy and pipeline evaluations short-circuit to Verdict::Deny. Recovery is a corrected bundle pushed through the hot-reload path. The path validates the replacement before an atomic swap.

Before a reload is accepted, a canary harness replays a frozen 32-fixture corpus against the new module; a failure leaves the prior epoch untouched. Publish is an atomic epoch swap: in-flight evaluations keep their original module snapshot while new calls use the new epoch. The reload metric chio_guard_reload_total labels by outcome: applied, canary_failed, rolled_back.

A post-publish rollback watchdog attaches to every accepted reload and trips after 5 consecutive error-class verdicts within 60 seconds: traps, fuel exhaustion, serialization failures, and other fail-closed backend errors. On trip it restores the prior module, emits chio.guard.reload.rolled_back, and writes an incident directory under ${XDG_STATE_HOME}/chio/incidents/<utc-iso8601>-<guard_id>-<reload_seq>/ holding incident.json and last_5_eval_traces.ndjson (redacted trace summaries only; request payloads are never persisted).

Digest Blocklist

A known-bad guard build can be pinned out of rotation by digest. The local blocklist lives at ${XDG_STATE_HOME}/chio/guards/blocklist.json. Engine::reload refuses a replacement module whose sha256:<digest> is listed and returns E_GUARD_DIGEST_BLOCKLISTED, and chio guard pull runs the same check against the pinned OCI manifest digest before it fetches or writes cache. Clear an entry with:

chio guard blocklist remove sha256:<digest>

Circuit-Breaker Recovery

The breaker transitions automatically. Open transitions to HalfOpen once reset_timeout has elapsed; HalfOpen transitions to Closed after success_threshold consecutive successes, and any failure while probing reopens it. The running service exposes no reset API: no admin endpoint or CLI subcommand forces the breaker Closed. Deployment recovery therefore follows provider recovery instead of a manual override. Callers that embed chio-guards as a library have one library API: CircuitBreaker::reset() is a public method that clears the failure window and returns the breaker to Closed for operator intervention.

Session Journal Loss

A SessionJournal is in-process state: a Mutex-guarded, capacity-bounded ring built via new, from_memory_budget, or with_caps. It has no persistence and no replay-from-receipts path, so it does not survive a kernel restart: a restarted kernel starts every session journal empty, and session-aware guards rebuild their view from new traffic. Signed receipts remain the durable audit record; the journal is the fast in-memory index guards read during a session, not a store to recover.

Worked Example: Tuned for a Flaky Provider

# The adapter block nests under a provider, not a standalone file.
# Path here: guards.threat_intel.safe_browsing.adapter
guards:
  threat_intel:
    safe_browsing:
      api_key: "sb-provider-key"        # required provider credential
      adapter:
        # Cache aggressively because the provider is unreliable.
        cache_ttl_seconds: 120          # default 60

        # Provider QPS is 50; leave 20% headroom.
        rate_per_second: 40             # default 20
        rate_burst: 40                  # default 20

        # Trip the breaker after five failures in the rolling window.
        circuit_failure_threshold: 5    # default 5 (unchanged)

        # Retry transient failures up to twice (three total attempts).
        retry_max_retries: 2            # default 3

The adapter block deserializes as ExternalAdapterPolicyConfig, which is deny_unknown_fields and exposes exactly five knobs: keys outside that set fail policy validation. Four move off their defaults here: cache_ttl_seconds 60 → 120 to absorb more of the provider's flakiness in cache; rate_per_second and rate_burst both 20 → 40 to track the provider's stated 50 QPS with headroom; and retry_max_retries 3 → 2 so a degraded service is not amplified. circuit_failure_threshold stays at its default 5.

Everything else the breaker and retry loop use: failure_window (60s), success_threshold (2), reset_timeout (30s), base_delay (100ms), max_delay (5s), jitter_fraction (0.25), the 1024-entry cache capacity, and the deny-on-open and deny-on-rate-limit verdicts, comes from the compiled adapter's built-in defaults and has no policy-YAML mapping. The backoff strategy is fixed to Exponential.

Next Steps

• Fail-Closed Semantics · the design contract behind these defaults
• External Adapters · adapter wiring, scoped tool patterns, and HushSpec authoring
• Observability · which signals to watch for early breaker telemetry
• External Guards · operator contract for the six bundled providers