Istio ext_authz

Chio runs as an Istio ext_authz provider. Every request through the mesh that matches the demo AuthorizationPolicy routes to the chio sidecar over gRPC; chio answers allow or deny; the mesh forwards (allow) or rejects (deny). Allowed responses carry an x-chio-receipt-id header that names the signed receipt the kernel produced.

Where the code lives

examples/istio-ext-authz/. The reference manifests target Istio 1.22+ on Kubernetes 1.28+. See ci-validation.md in the example for static-validation recipes (kubeconform, istioctl analyze) you can run without a live cluster.

What It Shows

The chio Envoy ext_authz adapter (chio-envoy-ext-authz) deployed as a cluster-wide Deployment in the chio-system namespace.
Chio registered under MeshConfig.extensionProviders with the gRPC backend at chio-sidecar.chio-system.svc.cluster.local:9091.
Three AuthorizationPolicy objects: a CUSTOM policy that opts matched workloads into chio, a DENY backstop for unauthenticated requests, and an ALLOW rule that lets kubelet probes through.
A demo workload in the agent-tools namespace that opts in via chio.protocol/secured=true.
A test harness that proves an authenticated POST gets HTTP 200 plus an x-chio-receipt-id header, and an unauthenticated POST gets 403 from the backstop.

Architecture

The mesh sidecar (Envoy) on the demo workload pod is configured with the envoyExtAuthzGrpc backend pointing at the chio sidecar Service. Every matched request is held by Envoy while it issues a gRPC CheckRequest against chio. Chio runs the kernel pipeline, signs a receipt, and responds with allow plus header mutations or deny. Envoy then forwards or rejects.

text

Client
   |
   v  POST /tools/hello (with x-chio-capability-token)
+---------------------+
| Envoy sidecar       |   <-- AuthorizationPolicy CUSTOM matches
| (demo-tool pod)     |       and routes ext_authz to chio
+---------------------+
   |  gRPC Authorization/Check (port 9091)
   v
+----------------------------+
| chio-sidecar Deployment    |   chio-system namespace
| chio-envoy-ext-authz       |   Service: chio-sidecar:9091/gRPC, 9090/HTTP
| (kernel pipeline + signer) |   /health on 9090
+----------------------------+
   |  CheckResponse: allow + x-chio-receipt-id header
   v
+---------------------+
| Envoy sidecar       |   forwards upstream
+---------------------+
   |
   v
demo-tool app (go-httpbin)

Prerequisites

Istio 1.22+ (the typed-header forwarding used here stabilized in 1.22).
Kubernetes 1.28+ (required for the security.istio.io/v1 and networking.istio.io/v1 GA APIs the policies use).
kubectl with cluster access; istioctl 1.22+; curl and awk on the workstation that runs the harness.
A dedicated chio Envoy ext_authz adapter image. The reference manifest uses ghcr.io/backbay-labs/chio-ext-authz:latest as a placeholder. Replace it with the adapter image you built and published. Do not use the generic ghcr.io/backbay-labs/chio-sidecar image: that is the HTTP sidecar and does not expose the gRPC Authorization/Check service on :9091.
A capability token for the demo workload (or the demo token chio accepts in shadow mode). Export it as CHIO_DEMO_CAPABILITY_TOKEN before running the harness.

Files

text

examples/istio-ext-authz/
  00-chio-sidecar-deployment.yaml   Namespace, Deployment, Service, ConfigMap, Secret
  01-meshconfig-patch.yaml          IstioOperator overlay; registers chio-ext-authz
  02-authorization-policy.yaml      CUSTOM (opt-in), DENY (backstop), ALLOW (probes)
  03-demo-workload.yaml             agent-tools namespace + go-httpbin demo
  README.md
  ci-validation.md                  kubeconform + istioctl analyze recipes
  test-harness.sh                   port-forward + curl harness

Step 1: Deploy the Chio Sidecar

bash

kubectl apply -f examples/istio-ext-authz/00-chio-sidecar-deployment.yaml
kubectl -n chio-system rollout status deploy/chio-sidecar --timeout=120s
kubectl -n chio-system get svc chio-sidecar

Expected Service output:

text

NAME          TYPE        CLUSTER-IP      PORT(S)             AGE
chio-sidecar   ClusterIP   10.96.42.7      9091/TCP,9090/TCP   20s

Smoke-test the adapter from inside the cluster before wiring Istio at it:

bash

kubectl run chio-smoke --rm -it --restart=Never \
  --image=curlimages/curl:8.9.1 -- \
  curl -sS http://chio-sidecar.chio-system.svc.cluster.local:9090/health

A 200 OK with body {"status":"ok"} confirms the sidecar is live.

Fail-closed by design

The Deployment exits non-zero when it cannot load its kernel config or policy bundle. Kubernetes restarts it but it never becomes ready, so Istio's ext_authz calls deny per failure_mode_allow=false. Chio outage means deny, not silent allow.

Step 2: Register Chio in MeshConfig

Pick the install path that matches how your mesh was provisioned.

Fresh install:

bash

istioctl install -y \
  -f examples/istio-ext-authz/01-meshconfig-patch.yaml

Existing mesh managed via IstioOperator:

bash

kubectl -n istio-system apply \
  -f examples/istio-ext-authz/01-meshconfig-patch.yaml
kubectl -n istio-system rollout restart deploy/istiod

The patch registers chio under meshConfig.extensionProviders:

examples/istio-ext-authz/01-meshconfig-patch.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: chio-ext-authz-meshconfig
  namespace: istio-system
spec:
  meshConfig:
    extensionProviders:
      - name: chio-ext-authz
        envoyExtAuthzGrpc:
          service: chio-sidecar.chio-system.svc.cluster.local
          port: 9091
          timeout: 0.25s
          includeRequestHeadersInCheck:
            - authorization
            - x-chio-capability-token
            - x-chio-session-id
            - x-request-id
          includeAdditionalHeadersInCheck:
            x-chio-source: istio-mesh
          statusOnError: "403"
          includePeerCertificate: true

Verify the provider was picked up:

bash

kubectl -n istio-system logs deploy/istiod | grep -i extensionprovider
istioctl proxy-config bootstrap -n istio-system deploy/istiod \
  | grep -A3 chio-ext-authz

Step 3: Deploy the Demo Workload and Policy

bash

kubectl apply -f examples/istio-ext-authz/03-demo-workload.yaml
kubectl -n agent-tools rollout status deploy/demo-tool --timeout=120s
kubectl apply -f examples/istio-ext-authz/02-authorization-policy.yaml
kubectl -n agent-tools get authorizationpolicies

Expected:

text

NAME                         ACTION     AGE
chio-tool-authorization       CUSTOM     5s
chio-deny-unauthenticated     DENY       5s
chio-allow-health-probes      ALLOW      5s

The CUSTOM policy opts in workloads labelled chio.protocol/secured=true. It matches POSTs to /tools/* and /invoke/* when either x-chio-capability-token or a Bearer ... Authorization header is present, and GETs on /tools/* when an Authorization header is present:

examples/istio-ext-authz/02-authorization-policy.yaml

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: chio-tool-authorization
  namespace: agent-tools
  labels:
    app.kubernetes.io/part-of: chio-protocol
spec:
  selector:
    matchLabels:
      chio.protocol/secured: "true"
  action: CUSTOM
  provider:
    name: chio-ext-authz
  rules:
    # Allow-rule #1: POSTs with a chio capability header
    - to:
        - operation:
            methods: ["POST"]
            paths: ["/tools/*", "/invoke/*"]
      when:
        - key: request.headers[x-chio-capability-token]
          notValues: [""]
    # Allow-rule #2: POSTs authenticating with a Bearer token
    - to:
        - operation:
            methods: ["POST"]
            paths: ["/tools/*", "/invoke/*"]
      when:
        - key: request.headers[authorization]
          values: ["Bearer *"]
    # Allow-rule #3: GETs on /tools/* with an Authorization header
    - to:
        - operation:
            methods: ["GET"]
            paths: ["/tools/*"]
      when:
        - key: request.headers[authorization]
          values: ["Bearer *"]

Confirm Envoy on the demo pod picked up the provider reference:

bash

POD="$(kubectl -n agent-tools get pod \
  -l app.kubernetes.io/name=demo-tool \
  -o jsonpath='{.items[0].metadata.name}')"
istioctl proxy-config listener -n agent-tools "${POD}" \
  --port 8080 -o json | grep chio-ext-authz

Step 4: Run the Harness

bash

export CHIO_DEMO_CAPABILITY_TOKEN="$(cat ~/.chio/demo.token)"
./examples/istio-ext-authz/test-harness.sh

The harness:

Opens a kubectl port-forward to svc/demo-tool:80.
Sends POST /tools/hello with x-chio-capability-token and asserts HTTP 200 with an x-chio-receipt-id header.
Sends the same request without credentials and asserts HTTP 403.

Expected tail output:

text

istio-ext-authz test-harness: PASS
  artifacts: .../examples/istio-ext-authz/.artifacts/<timestamp>
  allow receipt id: 01k6b1...-....
  deny status:      403

Manual curl Verification

bash

kubectl -n agent-tools port-forward svc/demo-tool 18080:80 &

# Allow path: chio evaluates and injects x-chio-receipt-id
curl -i -X POST \
  -H "x-chio-capability-token: ${CHIO_DEMO_CAPABILITY_TOKEN}" \
  --data '{"hello":"world"}' \
  http://127.0.0.1:18080/tools/hello

# Deny path: the DENY AuthorizationPolicy rejects before chio
curl -i -X POST --data '{}' http://127.0.0.1:18080/tools/hello

Allow response headers (order not significant):

text

HTTP/1.1 200 OK
x-chio-receipt-id: 01k6b1k3v...-0c7f
x-chio-policy-hash: sha256:...
x-chio-verdict: allow

Deny response:

text

HTTP/1.1 403 Forbidden
x-chio-denial-guard: IstioAuthorization

Smoke Assertions

The test harness asserts both directions match the contract. Verbatim from test-harness.sh:

test-harness.sh

# Allow request: capability + bearer token, POST /tools/hello
ALLOW_STATUS="$(
  "${CURL}" -sS \
    -o "${ALLOW_BODY}" -D "${ALLOW_HEADERS}" -w '%{http_code}' \
    -H "x-chio-capability-token: ${CAPABILITY_TOKEN}" \
    -H "authorization: Bearer ${CAPABILITY_TOKEN}" \
    -X POST --data '{"hello":"world"}' \
    "http://127.0.0.1:${LOCAL_PORT}/tools/hello"
)"

if [[ "${ALLOW_STATUS}" != "200" ]]; then
  fail "expected HTTP 200 on allow, got ${ALLOW_STATUS}"
fi

# Pull the receipt id out of the headers, case-insensitive
RECEIPT_ID="$(
  awk 'BEGIN{IGNORECASE=1} /^x-chio-receipt-id:/ \
       { sub(/\r$/, ""); print $2; exit }' "${ALLOW_HEADERS}"
)"

if [[ -z "${RECEIPT_ID}" ]]; then
  fail "allow response missing x-chio-receipt-id header"
fi

# Deny request: no credentials -> Istio DENY policy fires before chio
DENY_STATUS="$(
  "${CURL}" -sS -X POST --data '{"hello":"denied"}' \
    "http://127.0.0.1:${LOCAL_PORT}/tools/hello" \
    -w '%{http_code}' -o "${DENY_BODY}" -D "${DENY_HEADERS}"
)"

if [[ "${DENY_STATUS}" != "403" ]]; then
  fail "expected HTTP 403 on deny, got ${DENY_STATUS}"
fi

Decision rule

Use the Istio path when chio needs to govern every request through an existing service mesh and you already run Envoy sidecars. Pick hello-tool with the http-node middleware when the workload is a single Node service that does not need mesh-wide policy. The mesh path scales policy across many workloads at the cost of a per-request gRPC RTT to the chio sidecar; the middleware path keeps the decision in-process and skips the sidecar hop.

Troubleshooting

Symptom	Likely cause	Fix
All requests 403, including authenticated	Chio pod not ready or MeshConfig not reloaded	`kubectl -n chio-system get pods`; `istioctl proxy-config bootstrap ... \| grep chio-ext-authz`
Allow missing `x-chio-receipt-id`	`includeRequestHeadersInCheck` omitted or ext_authz in HTTP mode	Re-apply `01-meshconfig-patch.yaml`; confirm `envoyExtAuthzGrpc` is used
AuthorizationPolicy rejected at apply	API version mismatch (v1beta1 vs v1)	Cluster on Istio 1.22+ and serving `security.istio.io/v1`
Port-forward drops immediately	Pod not labelled `chio.protocol/secured=true`	`kubectl -n agent-tools get pod -l app.kubernetes.io/name=demo-tool --show-labels`
503 from demo pod	Istio sidecar injection disabled on `agent-tools`	`kubectl label ns agent-tools istio-injection=enabled --overwrite`

Teardown

bash

kubectl delete -f examples/istio-ext-authz/02-authorization-policy.yaml
kubectl delete -f examples/istio-ext-authz/03-demo-workload.yaml
kubectl delete -f examples/istio-ext-authz/00-chio-sidecar-deployment.yaml
# Optional: remove the extension provider from MeshConfig by editing the
# istio ConfigMap (or reinstall without 01-meshconfig-patch.yaml).

Where to read more

Envoy ext_authz for the full filter contract, gRPC vs HTTP modes, and the check protocol. Protect an API for the broader API-protection playbook.

PreviousOpenTelemetry NextEnvoy ext_authz reference