AI Test Harness · decks: Executives · Funding decision · Build or run it · Readers · Full deck · markdown source

Fund six weeks. Judge it on the evidence.

The case for an AI test harness, for CTOs, VPs of Engineering, and CISOs.

Chris Roadfeldt. Fifteen minutes.

1 / 13

The ask

A six-week pilot on two example projects already building on a supported build platform (Konflux, in the example), a named sponsor, a compute budget, and reviewers who will review what the harness produces.

At the end of six weeks I show you whether the tests are good enough to continue, measured against real breaking dependency upgrades, not against my opinion.

2 / 13

The problem, with numbers you can check

A mid-sized Go or Java service resolves several hundred to a few thousand packages. Engineers chose perhaps fifty.
The rest arrived as dependencies of dependencies. Nobody on our side has read or tested them.
A build platform such as Konflux already gives us hermetic builds, a bill of materials, and signed provenance for the build. It gives us nothing about whether any of that code works, or whether a new version still does what the old one did.

3 / 13

What the harness adds

Question	How the harness answers it
What does this code do?	AI-written tests, run and scored
Did it change from the version we had?	The same tests on old and new, compared
Does it fail safely?	Tests with malformed and hostile input
Are we exposed to what is already known to be wrong with it?	Tests aimed at each known vulnerability, producing draft VEX statements
Can we prove all of this later?	A signed record for every test, attached to the build
Does it fit how we already work?	A developer runs it while coding; the pipeline runs it on every change and proposes a test pull request. Tests arrive through pull requests and run with the normal suite

4 / 13

Why AI, and why now

Writing tests for code you did not write is slow and rarely prioritized, so it does not happen. Models are now good enough at reading unfamiliar code that the bottleneck moves from writing tests to verifying them, and verifying is something we automate: compile, run, mutate the code, compare old and new.

A test that survives that gauntlet is worth a reviewer's minute. One that does not is discarded before anyone sees it.

5 / 13

Why not just buy something

The best-known open source AI test generator is abandoned and copyleft-licensed.
The closest research tools ship only inside a commercial assistant, for one language.
Commercial tools cover one language each, do not run in our pipeline, and produce no provenance.
Nothing, open or commercial, does differential testing of a dependency upgrade for an application.

We would be buying a fraction of the capability and none of the evidence chain.

6 / 13

The evidence so far, on real code

One real pull request meant to fix four vulnerable packages, judged by the harness end to end:

Package	What the change did	Proven	Unproven
python-jose	updated	1 vulnerability closed	2
pyasn1	silently downgraded	1 vulnerability introduced	3
starlette	left alone	1 vulnerability live, upgrade path found	6

Three signed attestations. Nine of the blueprint's eleven goals at the time met (it has twelve today). Every unproven item is stated as unproven.

7 / 13

How it got there, and why that matters to you

The same package was run six times, changing one thing each run. The first four proved nothing. Each failure became a permanent, self-checked rule in the design, twenty-eight so far, and the harness refuses to run if any of them fails.

Run	One change	Proven
1 to 4	one-shot prompts, then tools	0
5	budget the tools, tell the model when it reached the fix	1 of 3
6	same model, faster server	2 of 3

The process improves by design, not by luck, and the record of it is public.

8 / 13

What it costs

Item	Pilot, six weeks	Steady state
People	2 engineers, part-time review from 2 teams	A small team; reviewer time scales with findings, not packages
Compute	Sandboxed runs for two services	Risk-weighted, budgeted per ecosystem, with alerts
Model usage	Metered, reported per run	Metered; a cheaper model for high-volume classification
Risk	Low: advisory only, nothing merges	Managed: only security findings block

No currency figure yet, on purpose: the pilot exists to produce one. Cost per work item is tracked from day one.

9 / 13

What could go wrong, and what the plan does about it

Risk	Mitigation
Tests are shallow	Mutation gate; differential runs; a public benchmark
Reviewers drown	Risk-weighted budgets, one-page packets, advisory by default
A malicious package escapes	No network, no secrets, disposable containers, scans before anything runs
The AI is manipulated by text in a package	Everything external is data; the agent has no tool that acts outside the sandbox
We claim provenance we cannot prove	Every attestation verified by policy; levels measured, not declared
Teams distrust AI tests	Every test labeled and human-approved; results published

10 / 13

What it returns

Fewer incidents from code we never looked at. The number I report quarterly: regressions and vulnerabilities caught that nothing else would have caught.
Faster, safer upgrades: a dependency bump arrives with a one-page packet within two hours.
Proof: every known vulnerability gets a VEX statement backed by a test or a reachability analysis.
A growing asset: tests that earn their keep join the standard suite.

11 / 13

The roadmap

gantt
    dateFormat  YYYY-MM-DD
    axisFormat  %b %Y
    section Phase 0 Pilot
    Two services, differential first        :p0, 2026-10-05, 6w
    section Phase 1
    Every PR, mutation, overlay repo        :p1, after p0, 6w
    section Phase 2
    Direct dependencies, CVE tests, VEX     :p2, after p1, 8w
    section Phase 3
    Transitive, reachability, enforcement   :p3, after p2, 10w

Dates are placeholders from a notional October start; durations get replaced by measured velocity after the pilot.

12 / 13

The decision

Fund six weeks. Judge the result on the benchmark and the reviewer feedback. Then decide about the next phase.

The blueprint, the working implementation, and every run's evidence are public on the project site.

13 / 13