AI Test Harness · decks: Executives · Funding decision · Build or run it · Readers · Full deck · markdown source

Blueprint, implementation, evidence

How the harness works, what it is built from, and what it has proven. For engineers, architects, and security analysts.

Chris Roadfeldt. Thirty minutes.

1 / 19

Three layers, one repository

Layer	Where	What it is
Blueprint	`docs/`, `blueprint/`	What the harness must do and why; the failure register; the schemas
Opinionated implementation	`harness/`	One way of doing it, for Python packages, with the capability map's choices fixed
Evidence	`examples/`	What the implementation produced on real code, kept exactly as produced

Two rules in order: the quality of a result comes first; then it is written so a person can read it.

2 / 19

The pipeline

flowchart LR
    V[0 Verify harness] --> I[1 Intake] --> A[2 Analyze] --> G[3 Generate] --> X[4 Execute] --> T[5 Triage] --> R[6 Review]
    R --> F[7 Feedback]
    F -.-> G
    F -.-> A

Stage 0 runs the failure-register checks, probes the sandbox, and probes the model. Any failure stops the run. No override.

3 / 19

One core, two lifecycles

	Developer's inner loop	Pipeline's outer loop
Starts	on the developer's branch, while working	on every pull request or merge, unattended
Runs	locally or ephemeral, same sealed sandbox	in the pipeline's sandbox
Produces	candidates and verdicts to iterate on	a packet and a test pull request
Gate	the code review	the test pull request
Afterwards	the suite runs the accepted tests	the suite runs the accepted tests

Stage 4 is validation: both versions, sealed, once. Regression is the suite's job, every change, and not a harness stage. Generated tests are untrusted until read, so the developer's loop uses the same sandbox, never a plain test runner. The CLI is the core as lifecycle A runs it; the step that opens the test pull request is what lifecycle B still needs.

4 / 19

Stage 1, intake: facts before anything runs

Resolve the full dependency graph for the project's own interpreter, inside its container image. pip on a different interpreter silently drops marker-gated packages; that was found the hard way.
Diff base against head; emit a CycloneDX SBOM.
One batch query to OSV for every package version, which doubles as the malicious-package gate.
Work list: one row per package with change, depth, reachability, and pre-flight state.

On the example PR this is where pip's silent downgrade of pyasn1 first appeared.

5 / 19

Stage 2, analyze: tools, not guesses

Per changed or vulnerable package: the public API surface from the AST, the contract diff with breaking flags, every first-party call site, advisories with fixed versions, the unified diff between versions as the "fix diff", and a risk score whose every contribution is written down.

Input	Weight
Reachable from first-party code	+35
Known vulnerabilities on this version	+25, +10 if rated
Breaking API changes	+15
Sensitive domain (auth, crypto, parsing)	+10

The score sets the generation budget. Stages 0 to 2 never call a model.

6 / 19

The sandbox, and how it is proven

Podman: no network, all capabilities dropped, no new privileges, read-only root, tmpfs work directory, memory, pid, cpu and time limits, environment cleared, packages installed offline from a prefetched wheelhouse, container removed afterwards.

A probe file runs inside the real sandbox at stage 0 and asserts each claim. It found the base image exporting a credential-shaped variable on the first run; the environment is cleared because of it.

7 / 19

Stage 3, generate: two variants, one set of gates

Fixed script: facts in as delimited data, one file out, compile, baseline run, repair with the error, cut failing tests, coverage gate.

Tool-using: the model gets search, read, list-API, and run-tests-on-both-versions, plus submit, under a budget. Reads are capped until a test has run; calls are reserved for a run and a submit; an identical result twice is reported as a blocked path; a failure carrying text the fix diff added is reported as "you reached the fix".

Every prompt, response, tool call, and result digest is written to disk. Tool results are data, never instructions.

8 / 19

Stage 4, execute: the gauntlet

Run on head. 2. Run again for flakes. 3. Coverage of the package. 4. Differential: run on the base commit's own graph, or on a resolved fixed candidate when the package did not change. 5. Mutation: a seeded, coverage-scoped sample of mutants overlaid into the sandbox; the passing tests run against each.

The verdict per test follows roles, not commit order: on a downgrade the new version is the vulnerable one. A CVE test counts only when it fails on the vulnerable version and passes on the fixed one.

9 / 19

Fixed candidates: when the change did not move the package

starlette was vulnerable at head and untouched by the PR. Pinning it to the fixed 1.3.1 does not resolve, because fastapi pins it. The harness relaxed fastapi's pin, resolved fastapi 0.141.1 with starlette 1.3.1, ran the tests against that, and wrote the upgrade path into the packet.

That is a finding a reviewer needs and a diff never shows.

10 / 19

Stages 5 and 6: triage and the packet

Triage is deterministic: the blueprint's classes with a confidence and a route; below 0.7 escalates to a person. An agent's claim of a defect counts only if stage 4 corroborates it.

The packet opens in plain terms: what the change did to the package, how many vulnerabilities were proven, how many remain unproven, what to do. Then the accepted tests as a patch in the overlay layout, findings with routes, a draft OpenVEX statement per vulnerability, and the artifact paths. Beside it, pull-request.md is what the test pull request carries, and the run's own README.md tells the whole story for a general reader: who, what, why, where, when, the decisions, the actions, and every file by the reader it is for.

11 / 19

Attestation, and the records behind it

A provenance record per accepted test, in the blueprint's schema.
The same facts as UDLM records: vulnerability, package, run, test evidence at the harness's provider class, draft VEX statement. Sealed with UDLM's own chain code.
An in-toto statement with the vetted test-result predicate whose subjects are the patch, the record file, and each evidence record's head. Signed as a DSSE envelope, verified.
An "unverified" list on every statement naming what the run could not back: development signing key, no mutation baseline, and so on.

12 / 19

The failure register: the harness tests itself first

Twenty-eight entries, each with the issue, why it matters, the cause, the detector, the automatic correction, and a self-check that runs at stage 0. Written check-first: the check fails on the old code and passes after the fix.

Fifteen of them, one line each; the full register is document 15.

Id	In one line
GF-003	catch-all assertions cannot fail; rejected
GF-006	tests imported names only the fixed version has; must collect on both
GF-013	the agent read source until its budget ran out; reads capped
GF-015	eleven identical failures in a row; reported as a blocked path
GF-016	on a downgrade the new version is the vulnerable one; roles from advisories
GF-019	reasoning streamed as content and looped; detected, thinking off
GF-020	a file that never collected on the baseline shipped as kept; now discarded
GF-021	a tool call cut off at the output limit poisoned the history; neutralised and reported
GF-022	one Go file that did not compile silenced the package; the rest run, its tests read as compile errors
GF-023	the agent submitted a file the sandbox had never built; submit now needs a clean run of that exact text
GF-024	one wrong line cost a Go file its nine good tests; the compiler's line cuts that test, the rest run
GF-025	a network hiccup on one model call ended a stage; the path is retried, the model's refusals are not
GF-026	a proven fix went unreported because the test's id was in capitals; the match ignores case
GF-027	a worse repair erased a working attempt; the last attempt that ran is kept, unused imports are pruned
GF-028	Go mutants were sampled from lines the tests never ran, so all survived; only executed lines count

13 / 19

The evidence, one pull request

Package	Change	Proven	Unproven	VEX draft
python-jose	3.3.0 to 3.4.0	1 closed	2	1 fixed, 2 under investigation
pyasn1	0.6.4 down to 0.4.8	1 introduced	3	1 affected with evidence
starlette	unchanged 0.41.3	1 live	6	7 affected, upgrade path

Mutation scores 0.42, 0.42, 0.0 against a 0.6 target: stated as not met. 76 UDLM records, all sealed, zero schema problems. Nine of eleven goals met.

14 / 19

The ladder: one variable per run

Run	Change	Proven	Time
1	baseline	0 of 3	22 min
2	weak-assertion gate, dependency imports, loop abort	0 of 3	15 min
3	fix diff and version-only symbols in the prompt	0 of 3	20 min
4	tool-using generation	0 of 3	60 min
5	read cap, fix-reached hint, blocked-path detection	1 of 3	55 min
6	same model on a two-GPU server	2 of 3	22 min

Every run is kept as produced. The register grew from each one.

15 / 19

Run it

cp harness/harness.example.toml harness/harness.local.toml   # endpoint, repo, interpreter, UDLM checkout
cd harness && python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/harness selfcheck --workdir out/run1                # stage 0
.venv/bin/harness intake   --base main --head HEAD --workdir out/run1
.venv/bin/harness analyze  --workdir out/run1
.venv/bin/harness generate --workdir out/run1 --select <pkg> --categories cve unit --mode agent
.venv/bin/harness execute  --workdir out/run1 --select <pkg>
.venv/bin/harness mutate   --workdir out/run1 --select <pkg>
.venv/bin/harness triage --workdir out/run1; harness packet --workdir out/run1; harness attest --workdir out/run1
.venv/bin/harness assess   --workdir out/run1

Or one command on any repository and two refs, which is what a trigger calls:

harness run --repo https://github.com/org/app.git --base main --head refs/pull/123/head --workdir out/pr-123 --propose

Examples for GitHub Actions, GitLab CI, Jenkins, Azure Pipelines and Tekton are under deploy/; a coding agent runs it through skills/ai-test-harness. Nothing about a machine, endpoint, or path is in the repository. Records carry labels and digests.

16 / 19

What is fixed, and what is a variable

Fixed: Tekton as the pipeline (on Konflux, in the example); Podman today and a Kata pod on the cluster; OSV; pip's resolver in the target image; the standard library AST; the harness's own mutator until a mutmut integration for installed packages exists; in-toto test-result; OpenVEX; UDLM records.

Variables, one at a time: the model and its serving; the generation variant; the budgets. The ladder is how a variable earns its place.

17 / 19

Known limits, stated

No model tried has proven a Go advisory: the 27B writes no Go that compiles, the 235B writes clean Go with one interface detail wrong per file. The evaluation table says so per model and prompt; a coder-class model is the next rung.
Mutation scores: 0.6 on Go, 0.25 to 0.4 on Python, against a 0.6 target; the packets say so.
Signing uses a development key; Trusted Artifact Signer replaces it in Konflux. The bot identity that opens requests is a placeholder until an account exists.
Go first-party targets and feedback inside the Tekton pipeline are not wired yet; propose is, behind a parameter and a secret.

18 / 19

What is next

A coder-class model as the next rung on the evaluation table, on Go first.
Go first-party targets: tests placed inside the module.
Feedback inside the Tekton pipeline, and a bot account with a signing key for the requests.
The remaining pipeline questions: what reaches the base branch untested, masked failures, unattested results.

The blueprint, the code, and every run are at the project site and repository.

19 / 19