Mutation Testing

What Mutation Testing Does

For each expression in the examined source, cargo-mutants generates a mutant: one mechanical edit such as flipping a comparison operator, swapping a boolean operator, deleting a negation, or replacing a function body with a type-appropriate stub. It rebuilds the crate with that one edit applied and reruns cargo test --workspace --exclude chio-cpp-kernel-ffi against it, the additional_cargo_test_args set in .cargo/mutants.toml (the C++ FFI crate ships through a separate vcpkg / conan lane the default workspace test run does not exercise).

Each mutant has one of four outcomes, recorded verbatim in outcomes.json and read by downstream script:

• CaughtMutant · a test failed. The mutant is killed; the suite noticed.
• MissedMutant · the test suite still passed. The mutant survived: a test gap.
• Timeout · the mutated build exceeded timeout_multiplier (3x the baseline, floored at minimum_test_timeout = 60 seconds). Scored as a survivor, the same as a miss.
• Unviable · the mutated code does not even compile. Excluded from scoring: scoreable = total - unviable, and the kill rate is caught / scoreable.

scripts/mutants-comment.sh buckets survivors by shape for its PR summary: deleted code, a swapped comparison or boolean operator, a deleted negation, a boolean or Result return stubbed to a fixed value, or a constructed value replaced with Default::default(). The script calls the mapping intentionally heuristic, since cargo-mutants exposes readable replacement text rather than a stable class enum across versions, but the shapes above are the actual edits a mutant can be.

How It Runs

The lane lives in .github/workflows/mutants.yml, and its trigger block determines when mutation testing runs:

on:
  # Mutation testing is heavy (cargo-mutants rebuilds per mutant), so it runs
  # nightly and on manual dispatch, not as a per-PR gate. The mutants-pr job
  # below is still gated on `github.event_name == 'pull_request'`, so without a
  # pull_request trigger it stays inert; only mutants-nightly runs on schedule.
  schedule:
    - cron: "0 5 * * *"
  workflow_dispatch:
    inputs:
      package:
        description: "Force a single crate (default: full matrix)"
        required: false
        type: string

There is no pull_request trigger. A mutants-pr job does exist, complete with --in-diff scoping against the PR base ref, a sticky PR comment, and survivor auto-filing. Because the workflow is not fired by a pull request, that job is currently inactive. The scheduled job is mutants-nightly.

mutants-nightly fires at 05:00 UTC, after the fuzz nightly (03:23) and the proptest nightly (04:23), sweeping six crates in parallel with cargo mutants --package <crate> --no-shuffle --timeout 600 --jobs 2 --json under a 240-minute per-crate job timeout. No --in-diff: this is a complete sweep of the examined modules. The job carries continue-on-error: true (it never blocks any check) and uploads mutants-out/<package> as a 30-day report. Both jobs share the 1,800 GHA-runner-minute, 30-day cap with the fuzz checks. The nightly job remains advisory if it exceeds that cap.

A companion workflow, mutants-fuzz-cocoverage.yml, runs at 05:45 UTC on a four-crate subset, re-deriving a fresh surviving-mutant set and replaying the accumulated libFuzzer corpus against each survivor: a different, adversarially generated oracle that sometimes catches mutants that the unit-test suite missed. The source documentation estimates a 5 to 15 percent cross-oracle reduction in missed mutants. The job is advisory and exits 0.

What It Measures

examine_globs in .cargo/mutants.toml scopes mutation to the trust-boundary modules of six crates. Generated mutants therefore affect modules that participate in decisions, rather than pure data or generated code:

• chio-kernel-core · the pure-compute verdict path (evaluate.rs, capability_verify.rs, scope.rs, receipts.rs, passport_verify.rs, guard.rs, normalized.rs).
• chio-policy · the HushSpec evaluator, compiler tree, conditions, detection, merge, resolve, validate, regex safety, and receipts.
• chio-guards · the fail-closed pipeline plus the boundary-enforcing guard set: shell command, path allowlist, egress and internal-network filters, secret leak, prompt injection, jailbreak detection, and roughly a dozen more.
• chio-credentials · lib.rs and trust_tier.rs: JWT VC, SD-JWT, OID4VCI, and OID4VP.
• chio-attest-verify · the Sigstore bundle parser and certificate-chain verifier.
• chio-anchor · anchor publication, discovery, and verification, including the Bitcoin, EVM, and Solana adapters.

A discovery quirk shapes two of those entries: cargo-mutants walks mod declarations, not include! macros, so chio-credentials/src/lib.rs (which pulls in thirteen trust-boundary files this way) and chio-policy/src/evaluate.rs (five more) are scoped at the umbrella file, which cargo-mutants can inspect; mutating it reaches its included files. Code outside examine_globs (platform adapters, the Kani harness files, tests, benches, fuzz entry points) is either covered by another lane or verification scaffolding, and each exclude_globs entry must carry a rationale: comment. scripts/check-mutants-rationale.sh fails closed without one, and CODEOWNERS gates any edit to the config.

Current State

The committed baseline at docs/fuzzing/trust-boundary-mutants-baseline.toml is the source for chio's public kill-rate figure:

generated_at = "2026-04-29"
tool = "cargo-mutants"
tool_version = "25.3.1"

[aggregate]
scope = "six-crate trust-boundary mutation baseline"
crate_entries = 6
listed_mutants_total = 2369
evaluated_mutants_total = 442
caught_total = 115
missed_total = 259
unviable_total = 67
timeout_total = 1
measured_kill_rate_excluding_unviable = 30.7
baseline_status = "aggregate complete from existing per-crate baselines; mixed full sweeps and bounded shards"

Read the arithmetic straight through: scoreable = 442 - 67 = 375 evaluated minus unviable, and 115 / 375 = 30.7% caught. The README banner rounds that to 31%, the same measurement at different precision. And listed_mutants_total is 2,369: the six-crate examined code generates more mutants than have been scored so far, consistent with baseline_status naming this a mixed aggregate of complete sweeps and bounded shards, not one full-workspace run, since each mutant reruns the entire test suite inside a four-hour per-crate nightly budget.

This is the figure Formal Assurance cites: the mutation-testing kill rate is about 30 percent, below the 80 percent target. The rate measures test sensitivity to selected implementation changes; it does not establish that those changes are detected. The drop and cancel-unwind path, which runs when a request is torn down mid-flight, has limited mutation coverage.

Whether this can ever fail a build is governed by releases.toml:

[mutants]
target_catch_ratio_percent = 80
activation_threshold_percent_per_crate = 65
required_consecutive_nightly_successes = 2
observed_consecutive_nightly_successes = 0
cycle_end_tag = ""

scripts/mutants-gate.sh remains advisory and exits 0 while cycle_end_tag is empty, which is its value today. The lane flips to blocking only after two consecutive nightly sweeps report a caught ratio at or above 80 percent (or the lower 65 percent per-crate floor, where set) on each of the six crates, and a CODEOWNERS-reviewed release PR writes the tag. releases.toml records each crate's kill rate as pending_full_sweep, so the per-crate scores required by this gate are not final.

Running It Locally

cargo install cargo-mutants --version '~25' --locked

# Scoped to the lines your branch actually changed. This is the same
# invocation the dormant mutants-pr job would use if it ever fired.
git diff origin/main...HEAD > /tmp/diff.patch
cargo mutants --in-diff /tmp/diff.patch

# Full sweep on one crate. Slow: cargo-mutants rebuilds and reruns the
# whole test suite once per generated mutant.
cargo mutants --package chio-kernel-core

Survivor triage

A surviving mutant resolves one of three ways, in order of preference:

• Add or strengthen a test. The default path. The mutant found a real gap; write the test that would have caught it.
• Refactor toward equivalence. If the mutant changes nothing observable, restructure the code so cargo-mutants prunes it, rather than skipping it.
• Skip with a rationale. Add the path to exclude_globs with a rationale: comment naming why the mutant is not a useful signal (equivalent code, generated code, a remote-process bridge better served by integration tests). CODEOWNERS review is required on the config file.

If the pull-request job is enabled, scripts/mutants-comment.sh would post one sticky per-crate comment (mutants, scoreable, caught, survivors, catch ratio, plus the top survivors inline), and scripts/mutants-autofile-issue.sh would open one fingerprinted issue per survivor beyond the PR cap, deduped on a SHA-256 of package, source location, verdict, and mutation text. While that job is inactive, the only report is the nightly mutants-out/<package> upload. For how chio triages the other formal gates once a failure surfaces, Failure Modes does not yet carry a mutation-specific section, but its Lean, Kani, and fuzz entries follow the same reproduce-classify-fix shape used above.

Next Steps

• Formal Assurance · the scope page that reports this lane's kill rate and target.
• Formal Assurance Overview · mutation testing's scope and limits among the assurance methods.
• Kani Harnesses · the bounded-model-checking lane sharing chio-kernel-core.
• Fuzz Infrastructure · cross-replayed against survivors nightly.
• Failure Modes · triage conventions for the other formal gates.
• Arena · a co-evolving adversary harness against the same guard pipeline.