How to read a run

Every run tells one story in three layers. Each layer is written by the harness from its own records, so they never disagree.

1. The story. Every run folder opens with a README.md: who, what, why, where, when; the outcome in plain terms; the decisions the harness made; the actions it took and the ones it did not take by design; the UDLM records in plain terms; and every file with the reader it is for. The same story is the top of each page under Runs.
2. The pull request. Tests reach a repository only through a pull request a person merges. packet/<package>/pull-request.md shows the text and the file list that pull request carries, whether or not one was opened; a review packet sits beside it with the verdict per test, the findings and the draft VEX statements.
3. The evidence. Everything else in the folder is the proof and the replay: sandbox runs, prompts and responses, signed statements, sealed records. Each file is labelled with its audience and its reason in the run's index and in document 10, so nothing is there without saying why.

The dashboard shows every run over time for three readers: leadership (what went through, what was proven, what a person acted on), engineering (which model and prompt set is fit for which job), pipeline owners (what the harness found about the CI it ran in). It is rendered from the same run indexes as the stories.

The nearest complete example: frc-scheduler-server, run 6, which went from a dependency bump through a test pull request to a person's merge and the realized records.

Executive summary

For: executives, board members, managing directors, and anyone who needs the point in five minutes.

The problem in one sentence

We test the code we write and trust the code we import, and most of what we ship is imported.

What I am proposing

An AI Test Harness: a system that uses AI agents to write tests for every piece of code that enters our products. It runs those tests in a locked-down environment, proves whether they are any good, and hands the results to a human for a decision. It covers our own code, the open source packages we choose to use, and the packages those packages pull in, down to any depth.

Why now

Three things changed at once.

1. Supply chain attacks are the dominant risk. The most damaging software incidents of the last several years came through dependencies, not through code our engineers wrote.
2. Regulators and customers now ask for proof. Frameworks such as SLSA and formats such as SBOM and VEX exist so that we can show, not just say, what we verified. Our build platform already produces most of that proof for the build step. It produces none of it for testing.
3. AI can now write tests that are worth running. Not perfect tests. Tests that, once we run them, score them, and let a human review them, are as good as the ones we write by hand and arrive in hours instead of quarters.

What it does, in plain terms

flowchart LR
    A[Code arrives] --> B[AI reads it and writes tests]
    B --> C[Tests run in a sealed box]
    C --> D[Results are scored and classified]
    D --> E[A person approves or rejects]
    E --> F[Signed evidence attached to the product]

• Code arrives. A pull request from our team, or a new version of a package we depend on.
• AI writes tests. Ordinary tests, tests that try to break the code, and tests aimed at known vulnerabilities.
• Tests run in a sealed box. No network, no secrets, thrown away afterward. Untrusted code cannot reach anything.
• Results are scored. We check that the tests would actually catch a bug, not just that they pass. We compare the new version against the old one to see what changed.
• A person approves. The harness never merges anything. It proposes. A reviewer decides.
• Evidence is attached. Every test carries a signed record of where it came from and who approved it. That record travels with the product and can be checked by anyone downstream.

It is used in two ways. A developer runs it while working, to get tests for the change in front of them, and the tests go in with their pull request. The pipeline runs it on every change, unattended, and proposes a test pull request for a person to read. Either way tests arrive through a pull request, and the accepted ones run with the normal test suite from then on.

What it is not

• It does not replace engineers or QE. It gives them evidence they do not have today.
• It does not merge code, publish security statements, or delete tests on its own. Every one of those is a human decision.
• It does not fix code. Fixes are a separate pipeline's job; this one hands it the failing test that proves the problem and tests whatever comes back. Keeping the two apart keeps the evidence independent of the fix.

What it costs and what it returns

The cost is compute, a small team, and reviewer time. The return is fewer incidents from dependencies we never looked at, faster and safer upgrades, and proof of testing that we can show a regulator or a customer. The harness exists and has run end to end on real repositories in the lab, including one dependency bump whose seven proposed tests a reviewer merged. The pilot is six weeks on two example projects, one Python and one Go, and answers whether the tests are good enough there before we spend more.

The one number that matters

Regressions and vulnerabilities caught by harness-generated tests that nothing else would have caught, reported quarterly. Every other metric is a means to that one.

What I need

A decision to run the pilot, a named sponsor, a compute budget for six weeks, and reviewers for the example projects' test pull requests. Details are in the case and the roadmap.

AI Test Harness Business Case

Blueprint Identity

Owner: Chris Roadfeldt

Elevator pitch: How do you know a change is safe when most of what you ship is code your team never wrote? The AI Test Harness writes the tests nobody will write by hand, for your own code and for every package you depend on, and makes every pull request arrive with its evidence: the tests that ran on both sides of the change, and a signed record of the results. The person at the gate decides with proof in hand instead of trust.

Problem Statement

Customer name: Any organization that ships software built on code it did not write: product teams, platform teams, and the security and compliance functions that answer for what they ship. The first pilot runs on two example projects, one Python and one Go; the first customer team is to be named with the sponsor.

Customer stakeholder: Whoever answers for the safety of a change and for the proof of it: the head of engineering, the product security lead, the platform owner. The pilot serves the person who approves the merge, and their measures come first: did every change arrive with evidence, how much review time did it cost, what did it catch that review alone would have missed.

Customer pain: Most of what any team ships is code it did not write. A mid-sized service resolves hundreds to thousands of packages; its engineers chose perhaps fifty, and nobody has read or tested the rest. Across a fleet the same unread package sits in every service, so one bad version lands everywhere at once; the XZ Utils backdoor arrived exactly this way. AI-assisted development widens the gap: teams with high AI adoption merge 98 percent more pull requests while review time per request rises 91 percent (Faros AI, 2025). Writing tests for code you did not write is never prioritized, so it does not happen.

Current workaround: A modern build platform says what is in the product, hermetic builds, a bill of materials, signed provenance, policy gates, but not whether any of it works. Teams bump dependencies on trust and read changelogs. Commercial AI test tools cover one language each, run outside the pipeline and produce no provenance; the best-known open one is abandoned. Nothing tests a dependency upgrade differentially for an application.

Impact of inaction: Regressions and vulnerabilities in code nobody looked at reach production and are found by customers. Every dependency bump stays a judgement call without evidence, so upgrades are delayed and known vulnerabilities stay open longer. Asked what we verified about the code we ship, we have no record to point at.

Impact of action: Every change arrives at the gate with tests that ran, a plain-terms verdict and a signed record. Every behaviour a change reaches is validated, handed to the team for their environment, or a stated gap with its reason; nothing is silent. Known vulnerabilities get a test that proves the fix and a draft VEX statement. Review time is the toil that remains, and it is measured on every run. The tests that earn their keep join the team's suite, so the asset grows with every run, and the evidence's fixed form lets approval be automated later within policy.

Proposed solution

Landing BUs: Trusted Software Supply Chain (the evidence chain), Red Hat OpenShift AI (model serving and the agent runtime), and the build platform and developer tooling teams. To be confirmed with the sponsor.

Aligned TDPs: To be named with the sponsor; the natural fits are the supply chain security and trusted AI plans.

Considered alternatives: IBM Research's designs ship only inside watsonx Code Assistant, for Java. The open generators are narrower or gone; the best known, Qodo-Cover, is unmaintained since 2025. Commercial tools cover one language each with no provenance. Buying would give a fraction of the capability and none of the evidence chain; the components underneath are open and maintained, and what did not exist was the assembly. Survey in the landscape.

Solution overview: A change arrives. The harness resolves the dependency graph at both versions, looks every package up for advisories, and scores risk to set a budget. Given facts from static analysis, a model writes unit, negative and vulnerability-targeted tests. Every candidate runs sealed on both sides of the change; a fix is proven only by a test that fails on the vulnerable version and passes on the fixed one, and mutation testing shows how firmly the tests hold. Triage classifies every verdict, the run writes its own story, and the tests reach the repository only through a pull request a person merges. The harness only tests: it never merges, fixes code, publishes or deletes. It is platform-agnostic and runs wherever the team's pipeline runs. The design is the blueprint; the positions settled in review are the decision record; a proof of concept has run it end to end on two real repositories, every run published.

Cost and noise, as measured on the example runs. These figures come from one lab and two models and are examples, not a forecast for another environment. A package change cost half a million to one and a quarter million prompt tokens and under fifty thousand completion tokens, plus a few CPU-minutes of sandbox and mutation time; the same tokens took 97 minutes of model time on one model and 8 on another, so model time is the cost and it follows the inference hardware. A platform owner prices it in their own tokens and throughput. The harness runs beside the pull request pipeline, adding nothing to its critical path. Nothing reaches a reviewer unless it built, passed on both sides, ran five times without disagreement and killed a mutant; the initial targets are a flake rate under 1 percent, false suspicious findings under 5 percent and reviewer acceptance above 70 percent (blueprint reference, section 10).

Timeline & Milestones:

1. Pilot, weeks 1 to 6. Two example projects, own code and direct dependencies, from the pull request. Exit criteria in the pilot plan.
2. First-party at scale, weeks 7 to 12. Every pull request, unattended; the test pull request as the standard path.
3. Direct dependencies, weeks 13 to 20. Every bump; CVE-targeted tests and VEX drafts on every advisory; SLSA Build L3 on the records.
4. Transitive dependencies, weeks 21 to 30. Reachability and risk scoring across the whole graph.
5. Continuous operation, week 31 on. Scheduled rescans, remaining ecosystems, upstream contributions; the extended goal is coverage of behaviour, not lines.

Durations assume two to four engineers and are replaced by measured velocity after the pilot.

Definition of Done: A product team can point the harness at a repository and a pull request and receive, without the harness team involved, a pull request that carries the tests, their results on both sides of the change and a signed record that the release gate verifies. On any new change every reached behaviour ends in one of the three states and nothing is claimed that was not proven. The reference jobs rerun on each release with no measure worse than the last. Every failure-register entry passes at the start of every run. It runs where the customer runs, in their build platform and through the shipped triggers. The full standard is decision D4 and D14 in the decision record.

References

• Faros AI, The AI Productivity Paradox, 2025.
• A survey of unit test case generation methods, Information and Software Technology 196, 2026.
• Meta, TestGen-LLM, 2024; IBM Research, ASTER, 2024.
• NIST, CVE-2024-3094: the XZ Utils backdoor.
• In this repository: the blueprint and its reference, the decision record, the pilot plan, the research notes with every industry source, and the evaluation table.

How it works

For: engineers, architects, and technical leaders who want the mechanism before the detail. Thirty minutes. The blueprint is the plan in three pages; its reference has everything this leaves out.

The shape of it

Every incoming change, regardless of where it came from, goes through the same seven stages. Two lifecycles wrap those stages, and they are worth naming before the stages themselves.

flowchart LR
    V[0 Verify harness] --> I[1 Intake]
    I --> A[2 Analyze] --> G[3 Generate] --> X[4 Execute] --> T[5 Triage] --> R[6 Review]
    R --> F[7 Feedback]
    F -.-> G
    F -.-> A

I run the same pipeline on three kinds of input, and the only thing that varies is the budget:

Two ways to run it

Creation and execution can be split as far as an organization wants. The harness's run is validation: both versions, sealed, once, producing the proof and the record. The suite's run is regression: head only, every change, by the CI that already runs the suite. The review gate sits between them, in code review or on the test pull request, and tests always arrive through a pull request. One rule holds in both loops: generated tests run before anyone has read them, so the developer's loop uses the same sealed sandbox, never a plain test runner on a laptop.

Either loop starts with one command on a repository by address and two refs: a pull request's base and head, a push's before and after, or a branch alone for a rescan. Document 13 shows the command and the triggers for GitHub Actions, GitLab CI, Tekton and a schedule.

Where it ends

The harness tests; it does not fix. A separate fix pipeline generates changes to production code, and this one feeds it. It hands over the failing test that proves a problem, a reproducer for a defect found on the way, the upgrade path when a fixed version exists, and the draft VEX statements. Whatever the fix pipeline sends back comes through here like any other change, and the test that motivated it is the first thing reported. The two are kept apart for four reasons. The evidence stays independent of the fix. Each pipeline's failure register stays about its own failures. Reviewers read one kind of change at a time. And the identity that proposes tests never needs the credentials that change production code.

Where it runs

The harness is not a new pipeline. It is one more integration test in the build platform a team already runs. The example throughout these documents is Konflux, Red Hat's open source software factory, which already builds hermetically, produces SBOMs, signs provenance at SLSA Build Level 3, and gates releases on policy. Any platform that gives the same four things fits the same way; the triggers under deploy/ show GitHub Actions, GitLab CI, Jenkins, Azure Pipelines and plain Tekton.

flowchart TB
    subgraph Konflux
        B[Build pipeline] --> S[Snapshot]
        S --> IS[integration-service]
        IS --> H[AI Test Harness PipelineRun]
        H --> C[Conforma policy check]
        C --> REL[Release pipeline]
    end
    HM[Hermeto: pinned deps + SBOM] --> B
    HM --> H
    H --> TR[Trustify: SBOM, vulns, VEX]
    H --> TAS[Trusted Artifact Signer]
    H --> OV[(Test overlay repo)]
    H --> PR[Review packet on the PR]

Stage by stage

0. Self-verification

Before anything else, the harness proves it is fit to run: every entry in its failure register has a passing self-check, the sandbox's isolation claims hold, and the model endpoint honors the settings the run will record. Any failure stops the run before intake. Document 15 is the register: for every way the generator has failed, the issue, the reason, the cause, the automatic correction, and the check that proves it on every run.

1. Intake

Hermeto has already resolved the complete dependency graph and produced the SBOM for the build. I diff that SBOM against the last known-good one and run cheap pre-flight scans (GuardDog, the OpenSSF Malicious Packages database, Capslock for Go). Out comes a work list: one row per package that changed, with its depth and whether our code can reach it.

2. Analysis

For each row, I gather facts with tools, not with the model. The public API surface from CodeLLM-Devkit or the language's own tooling. The contract diff from japicmp, gorelease, cargo-semver-checks, griffe, or API Extractor. The source diff. Our own call sites. Upstream's existing tests. Known vulnerabilities with their affected symbols. Static analysis output. Release notes, treated as untrusted text. Then a risk score decides how much generation budget this row gets.

Analysis also reads the repository's own CI definitions and counts what its existing suite reaches, because the pipeline is under test too. Where it is provably short of what validating a test needs, triage says so with the change that corrects it. The harness changes nothing. Document 13 lists what is checked.

3. Generation

The agent writes tests in the ecosystem's idiomatic framework, following the loop IBM Research published as ASTER: facts in, tests out, compile, run, repair, fill coverage gaps, mock what is external.

flowchart LR
    F[Facts from stage 2] --> P[Prompt]
    P --> M[Model writes tests]
    M --> C{Compiles?}
    C -- no --> P
    C -- yes --> RUN{Passes on baseline?}
    RUN -- no --> P
    RUN -- yes --> K{Adds coverage or kills a new mutant?}
    K -- no --> D[Discard]
    K -- yes --> KEEP[Keep as candidate]

Every test is written in the framework and layout the repository already uses and runs with the command the team already runs. No harness-specific runner. That is what makes the tests usable in existing CI/CD pipelines and standard operating procedures the day they are approved.

Six categories come out of this stage: unit tests, functional tests at our call sites, negative tests with hostile input, CVE-targeted tests, fuzz targets, and the harness code that makes all of it run.

4. Validation execution

This is the harness's own run, not the suite's. Everything runs in a sealed sandbox: pinned base image, Hermeto's dependency set mounted, no network, no secrets, hard resource limits, destroyed afterward. Kernel isolation is used where the cluster provides it; a test that needs a real kernel is handed to the team where it does not.

Then the gauntlet, in order: compile, run, re-run to catch flakes, measure coverage, mutate the code and see which tests notice, and run the same tests against the previous version to see what changed. Finally, the relevance check: which existing tests does this change make obsolete?

5. Triage

An agent classifies every failure and finding, with a confidence score and the evidence attached, in the style of Red Hat's sast-ai-workflow.

flowchart TB
    R[Result] --> Q{What is it?}
    Q --> TB[test-bug: regenerate, no human time]
    Q --> BC[behavior-change: documented, note it]
    Q --> UC[undocumented-change: flag for reviewer]
    Q --> DF[defect: open issue with reproducer]
    Q --> SC[security: Product Security]
    Q --> SU[suspicious: Supply Chain Security, block merge]
    Q --> OB[obsolete: propose retirement]
    Q --> RD[redundant: propose retirement]

Anything below the confidence threshold is escalated to a person instead of auto-routed.

6. Review

One packet per work item, posted where the build platform already reports. It holds a one-page summary, the tests as a patch against the overlay repository, coverage and mutation deltas, proposed promotions and retirements, draft VEX statements, logs, the replay manifest, and a Conforma attestation. In the pipeline's loop the harness opens a test pull request from it (harness propose: one branch under its own identity, the tests and the records in the overlay directory, the packet's summary as the pull request's text). The reviewer approves, edits, or rejects. Nothing merges without them, and what they accept runs with the suite from then on.

Two files make the review readable by anyone, not only the reviewer. The packet writes the pull request's text and file list (pull-request.md) whether or not a pull request is opened, so every run shows what a person would be asked to accept. And every run keeps a README.md at its top, rewritten after each stage. It says who, what, why, where and when, and gives the outcome in plain terms. It lists the decisions the harness made and the actions it took and did not take. It states each record as one plain sentence and names every file with the reader it is for. Reading the artifacts walks through both.

7. Feedback

Reviewer edits and rejections become labeled examples for prompt evaluation. In the pipeline's loop the harness reads the test pull request back after a person has decided: what landed unchanged, what was edited, what was dropped, who and when. Each accepted candidate's record becomes a requested and a realized record, and a signed statement names the merge commit. Tests that later catch a real regression are tagged. That tag is the number I report.

The same stage measures whether the model and the prompt are fit for purpose, because that is never assumed. Every prompt is a named, digested set under source control and every manifest says which set and which model produced a result. harness bench runs one job under several sets with everything else held, harness evaluate scores finished runs by what the reviewers and the sandbox said, and document 12 is that table for the example runs. A set or a model becomes a default only by winning it.

The test lifecycle

A generated test is not done when it passes. It has a life.

stateDiagram-v2
    [*] --> candidate: generated and validated
    candidate --> accepted: reviewer approves
    candidate --> [*]: rejected or discarded
    accepted --> standard: unique mutant kills, survived 2 bumps or caught a regression, owning team confirms
    accepted --> retired: obsolete or redundant, human approves
    standard --> retired: obsolete or redundant, human approves
    retired --> accepted: restored after safety re-run shows it still guards something
    retired --> [*]: dropped after one safety re-run

Every state change is recorded in the test's provenance record, which is signed and attached to the build attestation. Retired tests keep their record. Nothing is deleted.

Known vulnerabilities become evidence

For every CVE reported against a package version in the graph, the harness produces two tests and one draft statement.

sequenceDiagram
    participant TDA as Trustify Dependency Analytics
    participant H as Harness
    participant SB as Sandbox
    participant PS as Product Security
    participant TR as Trustify
    TDA->>H: CVE-X affects pkg v1.4.2, vulnerable symbol Parse
    H->>H: Is Parse reachable from our code?
    H->>SB: Exposure test at our call site with the known trigger
    SB-->>H: affected / not affected
    H->>SB: Fix-pinning test on v1.4.2 and v1.5.0
    SB-->>H: fails on 1.4.2, passes on 1.5.0
    H->>PS: Draft VEX with both tests as evidence
    PS-->>H: confirmed
    H->>TR: VEX statement
    H->>H: Promote fix-pinning test to standard suite

Provenance and SLSA

Every input the harness reads is already attested by the build platform (Konflux, in the example). Every output the harness produces is attested the same way, with an additional in-toto predicate carrying the test-evidence record. That record is a UDLM record. The test, the package, the vulnerability, the run, and the draft VEX statement are records in the Unified Data Lifecycle Model, each with its own tamper-evident head, and the signed statement names those heads as its subjects. The harness's own code, prompts, images, and models are built and attested the same way. Conforma verifies the chain and emits a Verification Summary Attestation.

What I build versus what I reuse

The capability map has the full table. The short version: I reuse the build platform, the SBOM tooling, the vulnerability store, the signing, the mutation engines, the reachability analyzers, the API diff tools, the pre-flight scanners, and the fuzz harness generator. I build the assembly, the generation and triage agents, cross-ecosystem differential testing, the relevance engine, the overlay repository, and the CVE-to-VEX path.

Creation and execution across languages and targets

Each language has an adapter that implements one interface, and each execution target has a provisioner. The language and target flows document has a diagram for each. The rule is: the harness produces one artifact, an OCI image plus a tmt plan, and the target only changes who provisions the machine that runs it.

AI Test Harness: the blueprint

Status: Draft v0.28, 2026-10-14. Owner: Chris Roadfeldt. For: engineers and architects deciding whether to build, run or adopt this. Three pages. Behind it: document 16 is the long form, with the section numbers the code and the other documents cite; document 15 is the failure register; document 14 lists what an implementation must do and the test that proves each.

Changes in v0.28: the behaviour worklist (stage 2). For every public function the parser lists the results a test should check: each return, each raise or error return, the fall-through, each parameter's type with the edges its domain implies, each decision with more than one condition. Each advisory is listed as a result that must never happen. The model reads it for the functions the application uses; the accounting counts it per reached behaviour as the denominator behaviour coverage will be measured against. Specification R27.

Changes in v0.27: mutation feedback (stage 3). Before a passing file is believed, the harness breaks the package where the file's tests ran and runs the file against each breakage. A breakage the tests let through goes back to the model once as a diff, and the manifest records what was caught after. Specification R26.

Changes in v0.26: the flake check runs the candidates about to be proposed as many times as the team sets, five by default (TestGen-LLM's figure), after the two runs every test gets; any disagreement quarantines. Specification R9 updated.

Changes in v0.25: testbed failure is its own status (stage 4), at the run level and per test. The environment's failure is never reported as the test's or the package's. The story opens with it, the accounting names it as the reason, and a CTRF report beside the results carries it as other. Specification R25.

Changes in v0.24: declared needs (stage 4). Every test says what it needs from its testbed, none, localhost, network or machine; the team says what the testbed provides. A test that needs more is withheld from the sealed run, recorded as skipped with its need, proposed to the team with the declaration, and its behaviour is accounted for your environment. A test that declared nothing and opened a connection is a defect in the test, and the verdict says so. Specification R24.

Changes in v0.23: per-test attribution for Go. The head run gives each passing test its own coverage profile and folds them into the coverage record as contexts. The accounting then applies the same rule in both ecosystems: the test that executed a behaviour must be the one that proved it.

Changes in v0.22: the accounting credits a behaviour with the private helpers it delegates to, the closure, from a new internals.json the extractors write for Python and Go. It attributes execution per test through coverage contexts recorded in the Python sandbox, and requires the test that executed a behaviour to be the one that proved it. Mutation sampling aims at the reached behaviours first. Specification R11 and R23 updated.

Changes in v0.21: the accounting (principle 15). Every behaviour the change reaches ends in exactly one state, validated, for your environment, or a stated gap with its reason, and nothing is silent. Triage writes it per package and rolled up; the story, the pull request text, the run index, the dashboard and the evaluation table carry it; assessment goal 13 checks that the counts sum and every gap has a reason. On the published runs it reads lower than "accepted", which is the point: see the dry run in document 18.

Changes in v0.20: the static analysis is named as implemented (reference, stage 2): the harness's own API extractors and diffs, the call-site scans, OSV, and now bandit, gosec and GuardDog in the image. Their findings reach the prompt as SECURITY FINDINGS, and a high-severity finding in the changed code is a security finding for a person. What the design names and the code does not yet run is listed beside it.

Changes in v0.19: cut to three pages for the reader evaluating it. Nothing is dropped: v0.18 is kept whole as document 16, the register as document 15, and the earlier change notes with them.

What it is

A pipeline in which AI agents write tests for every change entering the software supply chain and run them in a sealed sandbox against both the old and the new version. The tests are scored, and a signed, human-reviewable packet goes to a person. First-party commits, direct dependencies and transitive dependencies alike. The output is evidence: for each change, a reproducible answer to three questions.

1. What does this code actually do? (characterization)
2. Did its behavior change from the version we had before? (differential)
3. Does it do anything it should not, and is it exposed to what is already known to be wrong with it? (negative tests and CVE-targeted tests)

The harness tests. It never fixes, merges, publishes a VEX statement or deletes a test. A separate fix pipeline consumes what it hands over (failing tests with both runs kept, reproducers, the upgrade path, draft VEX statements), and the fix comes back through the harness like any other change (reference 2.4).

Scope

Out of scope: binary-only artifacts, performance testing, merging without a person, fixing production code. Ecosystems in priority order: Go, Python, Java and Kotlin, Rust, JavaScript and TypeScript, C and C++. One small adapter per ecosystem; the core is ecosystem-agnostic. It is platform-agnostic too: Konflux is the example build platform throughout these documents, and the harness runs wherever its triggers run, GitHub Actions, GitLab CI, Jenkins, Azure Pipelines or plain Tekton (deploy/).

Principles

Fifteen, numbered so the code, the goals and the other documents can cite them.

1. Generated tests are hypotheses until executed.
2. Tests must be strong, not just green. Mutation testing and differential runs tell them apart.
3. Incoming code is untrusted input. Source, docs, advisories and commit messages are data in every prompt; tests run sealed, with no network and no credentials.
4. Humans own the merge. The harness proposes; it never merges, publishes or changes policy.
5. Everything is reproducible. Model, prompts, tools, graph and image digest are recorded per run.
6. Cost is a first-class metric. Budget follows risk and stops when the next test is not worth it.
7. Reuse before building (documents 04 and 05).
8. Native first. Generated tests look and run like the team's own, with no harness present.
9. Generated tests are provenance, and candidates for the standard suite: each carries a signed record of source, model, prompt, version and approver.
10. The harness extends the chain of trust and never launders it. Its own code, prompts, images and models are attested like anything it tests.
11. The harness tests itself first. Every observed failure mode is a register entry with a detector, a correction and a self-check that runs before every run and fails closed.
12. The harness retires tests as well as writing them, by proposal with evidence, never by deletion.
13. Fitness for purpose is validated, never assumed. A prompt set or a model becomes the default only by winning a measured table.
14. The pipeline is under test too. A CI provably short of what validating a test needs is a finding with the correcting change, routed to its owners, never an edit.
15. Every reached behaviour is accounted for. Validated, for your environment, or a stated gap with its reason; nothing is silent, and nothing is implied that was not earned.

The pipeline

One core, eight stages, each writing the records the next one reads. The run's story and file index are rewritten after every stage.

Triage classes: test-bug, behavior-change, undocumented-change, defect, security, suspicious (blocks the merge), obsolete, redundant, and for the CI itself coverage-gap and pipeline.

Two lifecycles, one core. In the developer's inner loop the harness runs on the branch, in the same sealed sandbox, and the kept tests go into the developer's own pull request. In the CI outer loop it runs on every change unattended and opens a test pull request under a bot identity with signed commits and provenance. Dependency tests go to an overlay repository; first-party tests go to the repository under test, on the change's branch. In both, the merge is a pull request a person owns, and no generated test runs outside the sandbox before someone has read it. The gate reads two inputs, the pipeline's result and the harness's verdict, and either can hold the merge under the team's policy. The harness's run is validation execution; the suite's later runs are regression execution, not a harness stage.

Dependencies: where the budget goes

The risk score weighs reachability most, then change size and contract change, sensitivity, upstream signals, known vulnerabilities, pre-flight hits and history. The same tests on the old and the new version is the highest-value technique for dependencies and the first thing built. Cost is counted in tokens and CPU-minutes, the units that travel between environments: on the example runs a package change cost half a million to one and a quarter million prompt tokens and a few CPU-minutes besides, with model time following the hardware (reference, section 10).

Known vulnerabilities become evidence

For every advisory on a version in the graph: locate the vulnerable symbols from the advisory or the fix diff. Write an exposure test at our call site if our code reaches them, and a fix-pinning test that fails on the vulnerable version and passes on the fixed one. Draft a VEX statement: affected with the exposure test, not_affected with reachability evidence, fixed only on a proven test. Roles come from the advisories, not commit order, so a downgrade reports as exposure. Product Security confirms every statement; the harness never publishes one.

Test lifecycle and provenance

A test is a candidate until a reviewer accepts it, and accepted until the owning team promotes it to standard. Promotion means it kills a mutant no other test kills, has survived two version bumps or caught a regression, and asserts behavior we depend on. A test is retired only by a person's approval of a proposal with evidence, as a state, never a deletion. Dependency tests live in an overlay repository by ecosystem, package and version range; first-party tests live in the repository.

Every test carries a provenance record: a UDLM TestEvidence record, with SoftwarePackage, Vulnerability, Job and VexStatement around it, sealed with UDLM's tamper-evident head, and a signed in-toto test-result/v0.1 statement naming those heads as subjects. Endpoints, repositories and paths appear by label, name and digest only. SLSA targets: Build L3 for harness outputs, Source L3 for the overlay repository, Source L4 for the standard suite. The harness confers no level on what it tests.

Security rules

• Candidates run only sealed: no network, no secrets, read-only root, disposable. Kernel isolation (Kata) where the cluster provides it; where it does not, a test that declares it needs a real kernel is handed to the team, not run on a promise.
• A prompt injection yields at most a bad test, which validation discards, and is itself a suspicious finding.
• Signing keys live in the pipeline, never in the sandbox.
• Tests for undisclosed vulnerabilities stay restricted until disclosure; reproducers are rewritten, never copied.
• Hermeto (GPL-3.0) is invoked, never vendored; AGPL and unlicensed components are excluded.

Built here, reused from elsewhere

Reused: a build platform and its pipeline engine (Konflux and Tekton, in the example), Hermeto, Trustify, Conforma and Trusted Artifact Signer, each ecosystem's own resolver, runner, API-diff and mutation tools. Built here: the assembly and adapter interface, the generation and triage agents, cross-ecosystem differential testing, the overlay repository and its provenance, the relevance engine, and CVE-targeted generation with VEX drafts.

How it is judged

• Stage 0 passes: every register entry has a self-check; the register stands at twenty-eight.
• Every run is assessed against thirteen goals, each with its measurement and the file it came from.
• The reference jobs are rerun against the baseline in document 12.
• Document 14 lists the twenty-seven requirements and the test that proves each.
• Done is a standard of behaviour on work never seen before, every reached behaviour in one of three states and nothing claimed that was not proven, plus a regression check on the reference jobs, whose known answers are two-sided. The settled positions behind this are in document 21.

Targets that matter most: mutation score above 0.6 on sampled mutants, flake rate under 1%, reviewer acceptance above 70%, a CVE-targeted test for every known vulnerability, a signed record on every accepted test. The full metrics, risks, rollout, governance, open questions and worked example are in document 16.

Landscape: Existing Open Source Work Relevant to the AI Test Harness

Status: v0.1 Date: 2026-09-11 Companion to: 03-blueprint.md Decision record: 05-capability-map.md turns this inventory into one primary and one fallback per capability, including the orchestration, sandbox, and attestation choices.

This document records what already exists in the open source world that the harness can reuse, learn from, or must avoid. It was compiled from three research passes on 2026-09-11 covering Red Hat projects, IBM and IBM Research projects, and the wider ecosystem. Every project listed was verified by fetching its repository or paper page on that date. Star counts, release versions, and activity dates are as of that date.

1. Headline findings

1. Nobody has shipped the whole thing. There is no maintained open source project, from Red Hat, IBM, or anyone else, that takes arbitrary incoming source code plus its transitive dependencies and generates, executes, and validates unit and functional tests end to end. The pieces exist. The assembly does not.
2. IBM Research has published the closest blueprint but not the code. ASTER (unit tests), SAINT (service-level integration tests), and Sakura (multi-agent functional tests from natural language) are the most relevant papers of 2025 and 2026. The tools ship inside watsonx Code Assistant and are not open source. The one reusable IBM component is CodeLLM-Devkit, an Apache-2.0 program analysis SDK.
3. Red Hat's supply chain substrate is mature and is the natural home. Konflux, Hermeto, Conforma, Mobster, Trustify, and Trustify Dependency Analytics already resolve transitive graphs, produce SBOMs, prefetch dependencies for hermetic builds, and gate releases on policy. The harness should be a Konflux integration test step, not a new pipeline.
4. Red Hat's AI code-analysis work is in security triage, not test generation. sast-ai-workflow, Aegis AI, and Konveyor Kai are production-shaped patterns for "LLM agent reasons over code with confidence scoring." QualityFlow is the only Red Hat repo that generates test code, and it is early stage.
5. The best-known open LLM test generator is dead. Qodo-Cover (formerly Cover-Agent), the open reimplementation of Meta's TestGen-LLM, is AGPL-3.0 and its README now says "please fork." The maintained permissive alternatives are narrower: CoverUp for Python, TestSpark for Java and Kotlin inside an IDE, Pynguin as a hybrid search-plus-LLM tool for Python.
6. Mutation testing is solved per ecosystem. PIT, StrykerJS, cargo-mutants, mutmut, cosmic-ray, and gremlins all shipped releases in 2026. LLM-generated mutants are research grade only.
7. Differential testing of dependency upgrades exists only at distro scale. Debian autopkgtest, Rust Crater, Julia PkgEval, and R revdepcheck run reverse-dependency tests on upgrades. Nothing generic and cross-ecosystem does this for an application. This is the harness's clearest gap to fill.
8. Reachability is strongest for Go. govulncheck is the reference. OWASP dep-scan with atom is the broadest multi-language open option. CodeLLM-Devkit covers Java (including transitive jars), Python, and TypeScript.

2. Red Hat projects

2.1 AI-assisted test generation and code analysis

2.2 Supply chain, dependency analysis, hermetic builds, SBOMs

2.3 Testing infrastructure (no AI, likely substrate)

2.4 Platform context

• lightspeed-core/lightspeed-stack: assistant service over Llama Stack with RAG, MCP tools, shields. Developer Lightspeed generates test plans via chat, not test code.
• RamaLama (MIT) and Podman AI Lab: local model serving in containers. A candidate inference backend for the harness when open models are preferred.
• MCP servers, all Apache-2.0: kubernetes-mcp-server, openshift-mcp-server, trustify-mcp, insights-mcp, linux-mcp-server, konflux-devlake-mcp, jira-mcp-server. Red Hat also ships an MCP catalog and gateway on OpenShift AI.
• InstructLab archived Apr 2026. Its synthetic data work continues in Red-Hat-AI-Innovation-Team/sdg_hub. Not relevant to test generation.
• Project Lightwell (IBM and Red Hat, May 2026): AI-assisted vulnerability discovery, triage, and patch validation program. Commercial, not open source.

2.5 Searched for and not found at Red Hat

• Any open source project generating unit or functional tests for arbitrary incoming code with LLMs.
• Any repo named or described as an "AI test harness."
• AI features in tmt, Packit, or Konflux integration-service for test generation or selection. DevConf.US 2026 lists "AI in test failure analysis, test generation, and test selection" as a track theme, so interest exists.
• An open source release of the TPA exploit intelligence reachability feature.
• Public code for the Red Hat Research theses.

3. IBM and IBM Research

3.1 Test generation papers and tools

3.2 Code analysis for LLMs

3.3 Models and orchestration

3.4 Supply chain

• CycloneDX/sbom-utility and license-scanner: Go CLIs donated by IBM to CycloneDX. Apache-2.0. Validate, query, trim, patch SBOMs. No AI.
• IBM/sbom-utilities: browser SBOM analyzer. Tiny. No AI.
• Konveyor: IBM Research is a supporter alongside Red Hat. See section 2.1.

3.5 Searched for and not found at IBM

• Open source code for ASTER, SAINT, or Sakura.
• Any IBM LLM-based mutation testing or LLM fuzz-harness generation.
• Open source components of the watsonx Code Assistant test generation feature.
• Granite 3.x or 4.x dedicated code models.
• An IBM project combining LLMs with dependency reachability or SBOM analysis.

4. Wider open source ecosystem

4.1 LLM and AI test generation

Agent-based tools on Claude Code, Codex, Gemini CLI, or Aider: no widely adopted open source project wraps one of these specifically for test generation. What exists is skill and plugin directories, general orchestrators with a "tester" role, and small unverified individual projects. This is a gap the harness would fill.

4.2 Mutation testing

4.3 Fuzz harness generation with LLMs

4.4 Dependency behavioral and malicious-package analysis

4.5 SBOM, scoring, graph, vulnerability matching

4.6 API breaking-change detection

These are the cheapest "where did the contract change" signal to focus generated tests on changed surfaces.

4.7 Running tests against a new dependency version

No maintained open source tool generically installs a candidate dependency version and runs the consuming project's tests as a differential check across ecosystems. This is done ad hoc with CI matrices or at distro scale. It is the harness's clearest gap to fill.

4.8 Reachability

5. Gaps the harness would fill

1. A maintained, permissively licensed, multi-language, CLI-first, agentic test generator. None exists.
2. Generic cross-ecosystem differential testing of a consuming project against a candidate dependency version.
3. Test generation for transitive dependencies scoped by reachability. The pieces (CLDK, govulncheck, dep-scan) do reachability for vulnerabilities, not for test targeting.
4. An LLM-driven triage layer that classifies generated-test failures into test bug, behavior change, defect, security, and suspicious. sast-ai-workflow is the nearest pattern but targets SAST findings.
5. Rust fuzz harness generation with LLMs.
6. Anything of this shape packaged as a Konflux integration test or a tmt plan.

6. Implications for the plan

Reuse directly

• Konflux integration-service as the pipeline. Hermeto for hermetic dependency prefetch and SBOM. Mobster for SBOM lifecycle. Conforma to attest and gate. Trusted Artifact Signer to sign outputs.
• Trustify and Trustify Dependency Analytics for the transitive graph, direct-versus-transitive classification, and vulnerability data. trustify-mcp so the agent can query it.
• CodeLLM-Devkit for call graphs and reachability in Java, Python, TypeScript. govulncheck and Capslock for Go. dep-scan with atom for the rest.
• Per-ecosystem mutation tools: PIT, StrykerJS, cargo-mutants, mutmut or cosmic-ray, gremlins.
• API diff tools: japicmp, cargo-semver-checks, griffe, gorelease, API Extractor.
• GuardDog, Capslock, OpenSSF Malicious Packages as pre-flight suspicious gates. OpenSSF Package Analysis as the sandbox and behavioral-diff pattern.
• OSS-Fuzz-gen and Fuzz Introspector for C, C++, Java, Python fuzz harnesses.
• tmt to run OS-level functional tests on Testing Farm.
• prodsec-skills and harness-eval for the agent's own configuration.

Learn from, do not depend on

• ASTER, SAINT, Sakura for the generation loop design. QualityFlow for Red Hat-flavored agent orchestration. sast-ai-workflow for confidence-scored triage packaged as a Tekton task. CoverUp for flake handling and sandboxing. Qodo-Cover for the assured-improvement filter, reimplemented under a permissive license.
• Debian autopkgtest and Crater for reverse-dependency differential testing design.
• Log Detective for failure-log summarization.

Benchmark against

• BUMP for breaking dependency upgrades. TDD-Bench-Verified for fail-before, pass-after tests. Hamster for realism of generated tests. eval-dev-quality for generation quality.

Avoid

• Qodo-Cover, Mutahunter, packj (AGPL, and the first two are dormant). ChatUniTest, PromptFuzz (no license). Eclipse Steady, TestPilot, EvoSuite (unmaintained or version-locked). tackle-test-generator-cli (archived).

Licensing note

Hermeto is GPL-3.0-only. Invoking it as a separate tool in the pipeline is fine. Vendoring or linking its code into the harness would make the harness GPL. Everything else recommended for direct reuse is Apache-2.0, MIT, BSD, or ISC.

7. Sources

Primary sources were fetched on 2026-09-11 from GitHub repository pages and the GitHub REST API, arXiv abstract pages, research.ibm.com, research.redhat.com, developers.redhat.com, redhat.com blogs, openssf.org, pkg.go.dev, PyPI, docs.deps.dev, and project documentation sites. Individual URLs are linked inline above. Where a claim rests on a single secondary source it is marked as unverified in the text.

Capability map: what fills each capability, and what orchestrates it

For: engineers and architects choosing components. Companion to the landscape, which is the full inventory. This document is the decision: for every capability the harness needs, a primary choice, a fallback, and the reason. Research date 2026-09-11.

Two facts that reshape the choices

1. in-toto already has a vetted Test Result predicate. https://in-toto.io/attestation/test-result/v0.1 carries a result (PASSED, WARNED, FAILED), configuration descriptors, a URL, and lists of passed, warned, and failed tests. It is on the vetted list, not a draft. The harness emits this predicate, extended through its configuration descriptors, rather than inventing one. That answers open question 14 in the blueprint reference.
2. Konflux already runs an LLM inside a Tekton task for dependency PRs. konflux-ci/deptriage is a Go CLI that classifies semver impact and asks Gemini or Claude for impact analysis of dependency pull requests, with secret redaction and retry logic. It ships as an image with its own .tekton/ pipeline and was updated the day of this research. It is the precedent for the harness's intake and analysis steps and possibly an input to them.

Also worth knowing: Llama Stack was renamed OGX (Open GenAI Stack) in April 2026 and is GA in OpenShift AI 3.5 with the OpenAI-compatible Responses API. IBM's Agent Communication Protocol was absorbed into A2A, which joined the Linux Foundation's Agentic AI Foundation alongside MCP in August 2026. MCP and A2A are the only two agent protocols worth designing for.

The decision table

Orchestration options considered and set aside

Sandboxing options considered

IBM and Red Hat projects that inform this map

• konflux-ci/deptriage: LLM-in-Tekton dependency PR triage. Reuse its secret redaction, retry, and classification patterns; consider its output as harness input.
• konflux-ci/agent-plugins: Claude Code skills for Konflux. Packaging pattern.
• konflux-ci/kait-task: Tekton task calling an AI triage service on CI failures. Archived July 2026. Historical precedent only.
• Red Hat cloud-native agent blueprint (July 2026): Agent Sandbox plus OpenShell, Kuadrant MCP gateway, SPIFFE identity, MLflow and OTel tracing, TrustyAI guardrails. Adopted as the harness's runtime architecture, with Tekton steps mapped onto "agent as a workload" pods.
• IBM ALTK (Agent Lifecycle Toolkit): pre- and post-LLM and tool checks. Candidate source of output-validation components. License not yet verified.
• IBM ContextForge MCP Gateway 1.0 (April 2026): fallback to Kuadrant.
• Lightwell (IBM and Red Hat, May 2026): no public repository or tooling found as of the research date despite press references to an open source project. The harness sits squarely in Lightwell's stated remit of validating and testing fixes across open source at volume. Position it as Lightwell-aligned infrastructure and revisit when tooling is published.

Not found

• A Tekton Hub or Artifact Hub catalog task that invokes an LLM. The harness authors its own tasks.
• Firecracker as a VMM option in OpenShift sandboxed containers.
• A Red Hat support statement for gVisor on OpenShift.
• Public Lightwell tooling.

Workflows, process maps, and RACI

For: everyone who will touch the harness. Each item below has a trigger, a process map, a RACI, its inputs and outputs, and a service level. Diagrams are Mermaid and render on GitHub and in the slide deck.

Roles

RACI: R does the work. A owns the outcome, exactly one per row. C is consulted before. I is informed after.

Summary RACI across all workflows

0. Self-verification

Trigger. Every harness PipelineRun, before intake.

flowchart LR
    R[PipelineRun starts] --> SC[Run every self-check in the failure register]
    SC --> SB[Probe the sandbox: no network, no env, read-only, offline install]
    SB --> M[Probe the model endpoint: id, reasoning and sampling settings]
    M --> OK{All pass?}
    OK -- yes --> REC[selfcheck record written, attested with the run] --> S1[Stage 1]
    OK -- no --> STOP[Stop. No evidence produced. Harness Team notified]

Outputs. selfcheck.json: every check, its register id, pass or fail. Service level. Under two minutes. A failing check blocks the run; there is no override.

1. Intake and inventory

Trigger. Build platform snapshot ready (Konflux, in the example); PR opened or updated; lockfile or SBOM diff; scheduled rescan.

flowchart LR
    T[Trigger] --> H[Hermeto graph + SBOM]
    H --> M[Mobster: diff vs last known-good]
    M --> PF{Pre-flight gates:\nMalicious Packages,\nGuardDog, Capslock}
    PF -- hit --> SUS[Mark suspicious,\nnotify SCS, continue in sandbox only]
    PF -- clean --> WL[Work list:\npackage, versions, depth,\nreachable, preflight]
    SUS --> WL
    WL --> A2[Stage 2]

Inputs. Snapshot, previous SBOM, gate databases. Outputs. Work list, SBOM diff, pre-flight findings. Service level. Work list within 10 minutes of trigger.

2. Analysis and risk scoring

Trigger. Work list row.

flowchart TB
    W[Work list row] --> API[API surface: CLDK / language tooling]
    W --> DIFF[Source diff + API contract diff]
    W --> CS[Our call sites]
    W --> UT[Upstream tests]
    W --> VULN[Known CVEs + affected symbols]
    W --> SA[Static analysis]
    W --> NOTES[Release notes: untrusted]
    API & DIFF & CS & UT & VULN & SA & NOTES --> RS[Risk score]
    RS --> BUD[Generation budget]

Outputs. Fact bundle per row, risk score, budget. Service level. 15 minutes per row at depth 0 and 1.

3. Test generation

Trigger. Fact bundle with a nonzero budget.

flowchart LR
    FB[Fact bundle] --> P[Prompt: facts as data]
    P --> LLM[Model]
    LLM --> C{Compiles?}
    C -- no, retry < N --> P
    C -- no, retry = N --> LOG[Log and drop]
    C -- yes --> BASE{Passes on baseline?}
    BASE -- no --> P
    BASE -- yes --> IMP{Adds coverage or\nkills a new mutant?}
    IMP -- no --> DROP[Discard]
    IMP -- yes --> CAND[Candidate test + manifest entry]

Runs once per category: unit, functional, negative, CVE-targeted, fuzz, harness code.

Outputs. Candidate tests, JSON manifest, proposed fixes as separate patches. Service level. Within budget; never exceeds the per-item token and time caps.

4. Validation execution

Trigger. Candidate tests exist.

flowchart LR
    C[Candidates] --> SB[Sandbox: no network,\nno secrets, gVisor/Kata]
    SB --> S1[1 Compile]
    S1 --> S2[2 Run]
    S2 --> S3[3 Flake re-run]
    S3 --> S4[4 Coverage]
    S4 --> S5[5 Mutation, diff-scoped]
    S5 --> S6[6 Differential old vs new]
    S6 --> S7[7 Relevance check on existing tests]
    S7 --> RES[(results.json)]

Outputs. results.json, coverage, mutation score, differential diff, relevance findings. Service level. Depth 0 and 1 complete within 60 minutes; fuzzing time-boxed.

5. Triage

Trigger. results.json.

flowchart TB
    R[results.json] --> LOG[Summarize logs]
    LOG --> CLS[Classifier agent:\nevidence, verdict, confidence]
    CLS --> CONF{Confidence above threshold?}
    CONF -- no --> ESC[Escalate to human]
    CONF -- yes --> ROUTE{Class}
    ROUTE --> TB[test-bug: back to generation]
    ROUTE --> BC[behavior-change: note]
    ROUTE --> UC[undocumented-change: flag]
    ROUTE --> DF[defect: issue + reproducer]
    ROUTE --> SEC[security: PS]
    ROUTE --> SUS[suspicious: SCS, block]
    ROUTE --> OB[obsolete / redundant: retirement proposal]

Outputs. Classified findings, issues opened, merge block state. Service level. Classification within 15 minutes; suspicious acknowledged by SCS within one business day.

6. Review and merge

This is the review gate of the blueprint (reference section 5.0), and it has two shapes. In the developer's inner loop (workflow 16) the packet stays on the developer's branch and the gate is ordinary code review of their pull request. In the pipeline's outer loop the harness opens a test pull request under its own bot identity, with signed commits and the provenance attached, and the packet is the reviewer's brief. In both, tests arrive through a pull request, the harness never merges, and what is accepted runs with the suite from then on. Nothing about the harness's validation run is repeated by the suite; the suite runs the accepted tests on head, every change, like any other test.

Trigger. Review packet posted, on the developer's pull request or as a test pull request.

sequenceDiagram
    participant H as Harness
    participant PR as Pull request
    participant PT as Product Team reviewer
    participant SCS as Supply Chain Security
    participant PS as Product Security
    H->>PR: Review packet + attestation (on the developer's PR, or as a test PR)
    alt suspicious or security open
        PR-->>PT: Merge blocked
        SCS->>PR: Resolve or confirm suspicious
        PS->>PR: Resolve or confirm security
    end
    PT->>PR: Approve / edit / reject each test
    PT->>PR: Approve / reject retirements and promotions
    PT->>PR: Merge when policy allows
    PR-->>H: Decisions captured for feedback

Service level. Packet within 2 hours of trigger for depth 0 and 1. Review turnaround is the product team's normal PR SLA.

7. Feedback and prompt evaluation

Trigger. Reviewer decision recorded; regression caught by an overlay test.

flowchart LR
    D[Reviewer decisions] --> LAB[Labeled examples]
    REG[Regression caught] --> TAG[Tag the test]
    LAB --> EVAL[Eval suite]
    TAG --> METRICS[Quarterly metrics]
    EVAL --> PROMPT[Prompt change PR]
    PROMPT --> GATE{Benchmarks pass?}
    GATE -- yes --> MERGE[Merge prompt change]
    GATE -- no --> PROMPT

8. Promotion to the standard suite

Trigger. An accepted test meets the promotion criteria.

flowchart LR
    ACC[Accepted test] --> CRIT{Unique mutant kill AND\nsurvived 2 bumps or caught a regression AND\nasserts behavior we depend on?}
    CRIT -- no --> WAIT[Stay accepted]
    CRIT -- yes --> PROP[Harness proposes promotion]
    PROP --> R1[Reviewer 1: PT]
    R1 --> R2[Reviewer 2: PT or QE]
    R2 --> STD[Move to standard/, state = standard,\nrecord updated, attestation regenerated]

Two human approvals because the standard suite targets SLSA Source L4 and the harness is not a trusted person.

9. Test retirement

Trigger. Relevance check flags obsolete or redundant.

flowchart LR
    RC[Relevance finding] --> EV[Attach evidence:\nremoved symbol, deleted lines,\ncovering tests, mutant history]
    EV --> KIND{Target replaced,\nnot removed?}
    KIND -- yes --> REW[Send to generator as rewrite candidate]
    KIND -- no --> PROP[Propose retirement in packet]
    REW --> PROP
    PROP --> APP{Human approves?}
    APP -- no --> KEEP[Keep, note reason]
    APP -- yes --> RET[State = retired,\nrecord keeps reason]
    RET --> SAFE[One safety re-run on next bump]
    SAFE --> STILL{Still guards something?}
    STILL -- yes --> REST[Restore to accepted]
    STILL -- no --> DROP[Drop from execution, record kept]

Retirement proposals never block a merge.

10. CVE-targeted tests and VEX

Trigger. Trustify Dependency Analytics, OSV, or govulncheck reports a vulnerability for a package version in the graph.

flowchart TB
    CVE[Vulnerability reported] --> LOC[Locate vulnerable symbols:\nadvisory, fix commit, OSV data]
    LOC --> RCH{Reachable from our code?}
    RCH -- yes --> EXP[Exposure test at our call site]
    RCH -- no --> NA[Reachability record]
    EXP & NA --> FIX[Fix-pinning test:\nfails on vulnerable, passes on fixed]
    FIX --> VEX[Draft VEX in CSAF:\naffected / not_affected / fixed]
    VEX --> EMB{Public and fixed?}
    EMB -- no --> RESTRICT[Restricted visibility,\nembargo process]
    EMB -- yes --> PS[Product Security review]
    RESTRICT --> PS
    PS -- confirmed --> TR[Publish to Trustify]
    PS -- rejected --> REDO[Back to harness with reason]
    TR --> PROM[Fix-pinning test promoted to standard]

Service level. Draft VEX within 4 hours of the vulnerability appearing in the graph. Product Security confirmation per its own SLA.

11. Provenance and SLSA attestation

Trigger. Every harness PipelineRun.

flowchart LR
    RUN[Harness PipelineRun] --> PROV[Konflux SLSA Build provenance]
    RUN --> PRED[Test-evidence predicate:\nevery test produced, promoted, retired,\nevery draft VEX]
    PROV & PRED --> SIGN[Sign: Trusted Artifact Signer,\nkeys never in the sandbox]
    SIGN --> STORE[Attach to artifact, store in Rekor]
    STORE --> CONF[Conforma: verify provenance + predicate vs policy]
    CONF --> VSA[Verification Summary Attestation]
    VSA --> REL{Release policy}

12. Prompt and model change management

Trigger. A prompt file changes, or a model version is upgraded.

flowchart LR
    CH[Prompt or model change PR] --> BENCH[Run benchmark suite]
    BENCH --> CMP{Metrics within tolerance\nof current baseline?}
    CMP -- no --> REJ[Reject or iterate]
    CMP -- yes --> REV[Code review as for any source]
    REV --> MERGE[Merge, record model + prompt hash\nin every subsequent attestation]
    MERGE --> WATCH[Watch acceptance rate for 2 weeks]
    WATCH --> RB{Degraded?}
    RB -- yes --> ROLL[Roll back]

13. Adding an ecosystem adapter

Trigger. Roadmap phase or a product team request.

flowchart LR
    REQ[Request] --> TOOLS[Select tools per interface function:\nresolve, extract_api, api_diff, build,\nrun_tests, coverage, mutate, fuzz]
    TOOLS --> LIC[License and activity check]
    LIC --> IMPL[Implement adapter]
    IMPL --> IMG[Pinned sandbox image]
    IMG --> BENCH[Run on a benchmark repo]
    BENCH --> PILOT[Pilot on one product repo]
    PILOT --> GA[Enable by default]

14. Reusing an open source component

Trigger. Any new dependency of the harness itself.

flowchart LR
    C[Candidate component] --> L{License permissive?\nApache, MIT, BSD, ISC}
    L -- copyleft --> INV[Invoke as a separate tool only,\nnever vendor or link]
    L -- none --> REJ[Reject until licensed]
    L -- yes --> ACT{Release in the last 12 months?}
    ACT -- no --> ISO[Adopt only behind the adapter interface,\nplan replacement]
    ACT -- yes --> ADOPT[Adopt, record in landscape doc]
    INV --> ADOPT

15. Quarterly report

Trigger. End of quarter.

Content: regressions and vulnerabilities caught by harness tests that nothing else caught; VEX statements issued; standard suite growth; tests retired; cost per work item by ecosystem; SLSA level achieved; reviewer acceptance rate; open risks.

16. Developer inner loop

Trigger. A developer runs the harness on their branch, locally or on an ephemeral platform, while working on a change.

sequenceDiagram
    participant D as Developer
    participant H as Harness (local or ephemeral)
    participant SB as Sealed sandbox
    participant PR as Developer's pull request
    D->>H: run on the branch
    H->>SB: validate candidates on both versions
    SB-->>H: verdicts, coverage, provenance
    H-->>D: candidate tests and verdicts
    D->>D: keep, edit, drop; run again
    D->>PR: tests and provenance in the same pull request
    PR->>PR: normal code review (workflow 6)

Rules. The developer's run uses the same sealed sandbox as the pipeline; generated tests are untrusted until read, and a plain test runner on a workstation is not a substitute. Provenance records travel with the pull request. Whether the pipeline also runs its own loop on the same pull request is the organization's choice; when it does, the two sets of records are reconciled at the review gate.

Service level. Whatever the developer's own build takes; the harness adds validation time, not review time.

17. Pipeline findings: the CI under test

Trigger. Every run. Analysis reads the repository's CI definitions at the reviewed commit and counts what its existing suite reaches; triage states each provable gap as a finding with the change that corrects it (blueprint principle 14; document 13 lists the checks).

sequenceDiagram
    participant H as Harness
    participant PT as Product Team (pipeline owner)
    participant SCS as Supply Chain Security
    H->>H: read CI definitions; count the suite's reach
    H->>PT: pipeline and coverage-gap findings, each with a suggested change
    H->>SCS: security-class pipeline findings, for awareness
    PT->>PT: correct the pipeline, or record why not
    Note over H: the harness changes no pipeline

Rules. Fitting into the repository's CI is the default; fitting in silently is not. Findings are facts from the definitions, each with its rule, file and suggested change. Security-class findings (untrusted code reaching secrets or the network while tests run) are routed like other security findings; the rest advise. The harness never edits a pipeline; the owners decide, and the next run shows whether the gap closed.

Service level. Findings appear in the same packet as the run's tests; no extra cycle.

F1. Description fidelity (future)

Trigger. Any change with a human-written description.

flowchart LR
    D[Diff + API diff + call graph] --> AG[Agent describes the change,\ndescription withheld]
    AG --> CMP[Compare with stated description]
    CMP --> CLS{matches / incomplete /\nmismatch / undisclosed-sensitive}
    CLS --> MECH[Mechanical claim checks:\ntests added? no behavior change?\nfixes CVE? only this dep bumped?]
    MECH --> OUT[Finding in packet, feeds risk score]
    OUT --> ROUTE{mismatch or undisclosed on a dep?}
    ROUTE -- yes --> SCS[Max budget, route to SCS]
    ROUTE -- first-party --> AUTH[Back to author + second reviewer]

F2. Malicious or vulnerable change detection (future)

Trigger. Any change, any party, any depth.

flowchart TB
    CH[Change] --> SCD[Security-control delta:\nremoved checks, weakened TLS,\ndisabled tests, build/CI/install files]
    CH --> CAP[Capability delta:\nnew network, exec, fs, env,\nreflection, crypto]
    CH --> BG[Build-graph delta:\nnew packages at any depth]
    CH --> PROV[Provenance anomalies:\nsigning, first-time contributor\non sensitive files, cadence]
    CH --> FID[Description fidelity F1]
    SCD & CAP & BG & PROV & FID --> COMB[Combine with confidence]
    COMB --> IND[malicious-indicator finding]
    IND --> GATE{Deterministic signal corroborates?}
    GATE -- yes --> BLOCK[Block merge, route]
    GATE -- no --> ADV[Advisory, route]
    BLOCK & ADV --> WHO{Party}
    WHO -- third party and deeper --> SCS[Supply Chain Security]
    WHO -- first or second party --> PS[Product Security,\nreviewer other than author]

Roadmap

For: leadership and the team. What happens when, and how we know each phase is done.

gantt
    title AI Test Harness rollout
    dateFormat  YYYY-MM-DD
    axisFormat  %b %Y
    section Phase 0 Pilot
    Two services, depth 0 and 1, differential first     :p0, 2026-10-05, 6w
    section Phase 1 First-party at scale
    All PRs, mutation, flake, overlay repo, SLSA L2     :p1, after p0, 6w
    section Phase 2 Direct dependencies
    Bumps, pre-flight gates, CVE tests, VEX, SLSA L3    :p2, after p1, 8w
    section Phase 3 Transitive
    Reachability, risk scoring, full graph, enforcement :p3, after p2, 10w
    section Phase 4 Continuous
    Scheduled rescans, remaining ecosystems, upstream   :p4, after p3, 12w

Dates are placeholders from a notional October 2026 start. Durations are estimates for a team of two to four engineers and will be replaced by measured velocity after Phase 0.

Phase 0: Pilot, weeks 1 to 6

Goal. Prove that generated tests are strong enough to be worth a reviewer's time, on real code.

• Two example projects, one Go and one Python, both already building on a build platform (Konflux, in the example pilot).
• Our own code (depth 0) and direct dependencies (depth 1) only.
• Build the differential step first and run it against BUMP, the public benchmark of about 570 real breaking Java dependency upgrades.
• Unit, functional, negative and CVE-targeted tests, with mutation on the lines the tests executed: all already implemented and measured in the lab; the pilot measures them on the example projects.
• Triggered from the pull request (document 13). Review packets posted as PR comments; the test pull request opened in the pipeline's loop.
• Signed in-toto statements and sealed UDLM records per run, already produced by the pipeline, not by hand.
• The pipeline under test too: what each pilot service's CI does with untrusted code while tests run, and what its existing suite reaches (blueprint principle 14, questions 2 and 3); questions 1, 4 and 5 follow in Phase 1.
• Model and prompt fitness read from the evaluation table (document 12) before any default changes.

Exit criteria. Generated tests build more than 80 percent of the time after retries. At least one real finding. Reviewer feedback collected from both teams. BUMP catch rate measured and reported.

Phase 1: First-party at scale, weeks 7 to 12

Goal. Run on every PR in the pilot repositories without anyone pressing a button.

• Triggered by the build platform's integration step (Konflux integration-service, in the example).
• Mutation testing and flake detection added. Weak and flaky tests never reach a reviewer.
• Test overlay repository, provenance manifest, and in-toto predicate established.
• First version of the relevance engine: tests whose target symbol or code path was removed are proposed for retirement.
• SLSA Build L2 (signed with Trusted Artifact Signer) and Source L2 for the overlay repository.

Exit criteria. All metrics within 20 percent of target. Every accepted test carries a signed record.

Phase 2: Direct dependencies, weeks 13 to 20

Goal. Every dependency bump in the pilot repositories arrives with evidence.

• Differential testing and generation on every version change.
• Pre-flight gates (GuardDog, OpenSSF Malicious Packages, Capslock) and the suspicious-finding merge gate.
• CVE-targeted tests and draft VEX statements, every draft reviewed by Product Security.
• OSS-Fuzz-gen for packages flagged sensitive.
• Java and Rust adapters. CodeLLM-Devkit evaluated for Java reachability.
• Promotion to the standard suite goes live. Relevance engine adds behavior-change and redundancy detection.
• SLSA Build L3, Source L3 for overlays, Source L4 for the standard suite. Conforma emits Verification Summary Attestations. Policies advisory.

Exit criteria. Every direct dependency in a pilot release has at least one accepted test. Every known vulnerability has a confirmed VEX statement. Product Security confirms more than 80 percent of drafts without rework.

Phase 3: Transitive dependencies, weeks 21 to 30

Goal. Cover the full graph at a cost we can sustain.

• Reachability analysis and risk scoring live, fed by Trustify Dependency Analytics.
• Characterization snapshots for every package in the graph.
• Risk-weighted generation for depth 2 and deeper. CVE-targeted tests at every depth.
• Cost controls and budget alerts per ecosystem.
• Conforma policies move from advisory to enforced.

Exit criteria. Full-graph run completes within budget on both pilot services. Cost per work item reported. No release blocked by a false positive for four consecutive weeks.

Phase 4: Continuous operation, week 31 onward

• Scheduled full-graph rescans independent of PR activity.
• Remaining ecosystem adapters: JavaScript and TypeScript, C and C++. tmt plans for OS-level functional tests on Testing Farm.
• Upstream contribution workflow for high-value overlay tests.
• Quarterly report: regressions caught, VEX statements issued, cost, standard suite growth.

Documented for later, not scheduled

Two capabilities are designed into the architecture but not on this timeline. They share the harness's intake, sandbox, and provenance, and they are better built once the harness's own chain of trust is established.

• Description fidelity. Checking that a pull request or release note describes what the change actually does, with the description withheld from the AI until it has described the diff itself.
• Malicious or vulnerable change detection across first-party, second-party, third-party, and transitive code, including indirect changes to build scripts, CI configuration, tests, and lockfiles.

Both are specified in the blueprint reference, section 16.

What "done" means

The harness is done when a dependency bump at any depth, in any supported ecosystem, produces a signed, human-reviewed packet that says what changed, whether we are exposed to anything known, and which tests now guard it. The time it takes follows the inference hardware. And it is done when a quarterly report can show what that evidence caught.

Glossary

For: anyone. Plain-language definitions of the terms used in this project.

Harness creation and execution flows by language and target

For: engineers building or operating adapters. This is the document that says exactly what happens for Go versus Rust, and on a Kubernetes pod versus a bare-metal machine.

The contract that makes this manageable

Every language adapter produces the same artifact, and every execution target consumes it. That is the whole trick. The adapter knows the language. The provisioner knows the machine. Neither knows the other.

flowchart LR
    subgraph Adapter[Language adapter]
        R[resolve_graph] --> E[extract_api] --> D[api_diff] --> B[build] --> GEN[generate tests]
    end
    GEN --> ART[(Harness artifact:\nOCI image + tmt plan\n+ expected-results manifest)]
    ART --> P{Execution target}
    P --> K8S[Kubernetes / OpenShift pod]
    P --> POD[Podman container]
    P --> VM[Virtual machine]
    P --> BM[Bare metal]
    K8S --> RES[(results.json, coverage,\nmutation, logs, attestation)]
    POD --> RES
    VM --> RES
    BM --> RES

The harness artifact is:

Results come back in one schema regardless of target, and the attestation names the target class and the provisioner so that provenance is complete.

Choosing a target

flowchart TB
    S[Work item] --> Q1{Needs hardware, specific\narch, or measured performance?}
    Q1 -- yes --> BM[Bare metal via Testing Farm]
    Q1 -- no --> Q2{Needs a full OS: systemd,\nkernel modules, SELinux,\nRPM install, multiple users?}
    Q2 -- yes --> VM[Virtual machine via Testing Farm]
    Q2 -- no --> Q3{Running inside the\nKonflux pipeline?}
    Q3 -- yes --> K8S[Kubernetes pod with gVisor or Kata]
    Q3 -- no --> POD[Podman on a runner or a developer machine]

Creation flows by language

Each flow below is the adapter's implementation of the interface in blueprint/adapter-interface.md. Tools marked "verify" were not in the landscape research and need confirmation before adoption.

Go

flowchart LR
    R[Hermeto gomod\n+ go mod graph] --> E[gopls symbol export\n+ go doc]
    E --> D[gorelease / apidiff]
    D --> RCH[govulncheck reachability\n+ Capslock capabilities]
    RCH --> G[Generate: testing + testify,\nnative fuzz targets]
    G --> B[go vet, go build -mod=vendor]
    B --> ART[(Artifact)]

Python

flowchart LR
    R[Hermeto pip\n+ pipdeptree] --> E[griffe API dump\n+ CLDK Jedi/PyCG call graph]
    E --> D[griffe check]
    D --> RCH[CLDK reachability\n+ dep-scan]
    RCH --> G[Generate: pytest,\nhypothesis properties,\nAtheris fuzz targets]
    G --> B[Offline venv from\nprefetched wheels, import check]
    B --> ART[(Artifact)]

Java and Kotlin

flowchart LR
    R[Hermeto maven or\nGradle offline cache] --> E[CLDK codeanalyzer-java:\nsymbol table + call graph\nover source and jars]
    E --> D[japicmp on old vs new jar]
    D --> RCH[CLDK reachability\nacross transitive jars]
    RCH --> G[Generate: JUnit 5,\nMockito, Jazzer fuzz targets,\nRandoop regression oracles]
    G --> B[mvn -o or gradle --offline,\ncompile + test-compile]
    B --> ART[(Artifact)]

Rust

flowchart LR
    R[Hermeto cargo\n+ cargo metadata] --> E[rustdoc JSON]
    E --> D[cargo-semver-checks]
    D --> RCH[dep-scan atom\nreachability]
    RCH --> G[Generate: cargo test,\nproptest properties,\ncargo-fuzz targets]
    G --> B[cargo build --offline,\ncargo clippy]
    B --> ART[(Artifact)]

JavaScript and TypeScript

flowchart LR
    R[Hermeto npm / pnpm / yarn] --> E[API Extractor report\n+ CLDK ts-morph]
    E --> D[API Extractor diff]
    D --> RCH[CLDK reachability\n+ LavaMoat capability policy]
    RCH --> G[Generate: Vitest or Jest,\nfast-check properties,\nJazzer.js fuzz targets]
    G --> B[tsc --noEmit,\noffline install from cache]
    B --> ART[(Artifact)]

C and C++

flowchart LR
    R[Hermeto generic or rpm\n+ mock buildroot] --> E[clang AST export\n+ CLDK clang analyzer]
    E --> D[libabigail abidiff]
    D --> RCH[Fuzz Introspector\nreachability]
    RCH --> G[Generate: GoogleTest or Catch2,\nlibFuzzer targets via OSS-Fuzz-gen]
    G --> B[CMake or Autotools or\nrpmbuild in mock, offline]
    B --> ART[(Artifact)]

C and C++ is the ecosystem most likely to need the VM and bare-metal targets, because so much of it is RPM-delivered and kernel-adjacent.

Execution flows by target

Kubernetes and OpenShift (default)

sequenceDiagram
    participant IS as integration-service
    participant PR as Harness PipelineRun
    participant TR as Tekton TaskRun
    participant POD as Sandbox pod (gVisor/Kata)
    participant REG as Internal registry
    participant COL as Collector task
    IS->>PR: Snapshot ready
    PR->>REG: Pull harness artifact by digest
    PR->>TR: Execute task with RuntimeClass, deny-all NetworkPolicy, no SA token
    TR->>POD: Start, mount artifact read-only, emptyDir for output
    POD->>POD: compile, run, flake re-run, coverage, mutation, differential
    POD-->>TR: results.json + logs + coverage (emptyDir)
    TR->>COL: Hand off results
    COL->>COL: Classify, build attestation, sign via TAS
    COL-->>PR: Review packet, attestation
    Note over POD: Pod deleted. Nothing persists.

Controls: RuntimeClass kata where the cluster provides it, by admission policy; NetworkPolicy deny-all on the namespace; automountServiceAccountToken: false; read-only root filesystem; CPU, memory, and activeDeadlineSeconds limits. The signing step runs in a separate task that never shares a pod with test execution, which is what SLSA Build L3 requires.

Podman

flowchart LR
    A[Pull artifact by digest] --> B["podman run --rm --network=none\n--read-only --cap-drop=all\n--userns=auto --runtime=runsc or krun\n--memory --cpus --timeout"]
    B --> C[Runner script executes the tmt plan]
    C --> D[Results written to a mounted output dir]
    D --> E[Optional: collector container\nclassifies and, on CI runners, signs]

Podman is the developer's reproduction path. A review packet includes the exact podman run line and the artifact digest, so any engineer can replay the run on a laptop and get the same results.json. On CI runners outside the build platform the same image runs with the same flags and the collector signs; on a laptop there is no signing and the attestation is marked replay, unsigned.

Virtual machine (Testing Farm)

sequenceDiagram
    participant PR as Harness PipelineRun
    participant TF as Testing Farm API
    participant AR as Artemis provisioner
    participant VM as Fresh VM (RHEL / Fedora / CentOS Stream)
    participant COL as Collector task
    PR->>TF: Submit tmt plan + artifact digest + target class
    TF->>AR: Provision from golden image, arch, distro
    AR->>VM: Boot, attach, pull artifact from internal registry
    VM->>VM: Disable outbound network, run tmt plan
    VM-->>TF: results.json, coverage, logs, journal
    TF-->>PR: Results
    PR->>COL: Classify, attest, sign
    Note over VM: VM destroyed. Snapshot not kept.

Use it for RPM-delivered packages, for anything that touches systemd, SELinux, kernel interfaces or multiple users, and for distro matrix runs (RHEL 9, RHEL 10, Fedora, CentOS Stream). tmt's provision step selects the image; the plan is the same one that ran on Kubernetes.

Bare metal (Testing Farm, Beaker pool)

sequenceDiagram
    participant PR as Harness PipelineRun
    participant TF as Testing Farm API
    participant BK as Beaker
    participant HW as Physical host
    participant COL as Collector task
    PR->>TF: Submit tmt plan + artifact digest + hardware requirements
    TF->>BK: Reserve host matching arch, device, or performance class
    BK->>HW: Reprovision from clean image
    HW->>HW: Pull artifact, isolate network, run tmt plan
    HW-->>TF: results.json, coverage, logs, hardware telemetry
    TF-->>PR: Results
    PR->>COL: Classify, attest, sign
    BK->>HW: Wipe and return to pool

Use only when the risk score or the package class demands it: drivers, firmware interfaces, accelerator and GPU libraries, architecture-specific code, and packages whose failure mode is performance. Every bare-metal run is logged with its hardware identity in the attestation, and the wipe is verified before the host returns to the pool.

Language and target matrix

What the adapter interface guarantees

Adding a language means implementing eight functions: resolve_graph, extract_api, api_diff, build, run_tests, coverage, mutate, fuzz. Adding a target means implementing one provisioner that accepts a tmt plan and an artifact digest and returns results in the common schema. The pipeline, the agents, the triage, the provenance, and the review packet do not change for either.

Reading the artifacts

For: anyone handed a run of the harness. One section per role; each names the one file to open first, what it says, and what to do with it. A run is a directory; every stage writes into its own subdirectory, and every number in a packet points at a file in that directory.

The shape of a run

<run>/
  README.md                the story: who, what, why, where, when; outcome, decisions, actions; the records in plain terms
  run.json                 what was run, with what tools, on which repository (by name, never a path)
  run-index.json           what the run holds: stages, packages, headline numbers, and where every file is, each labelled with its reader
  selfcheck/               stage 0: every register check, the sandbox probe, the model probe
  intake/                  the dependency graph at base and head, the SBOM, known vulnerabilities, the work list
  analyze/<package>/       facts: API surface and diff, call sites, advisories, the risk score, the fix diff; internals.json, the private helpers; behaviours.json, the worklist of results to check
  generate/<package>/      the candidate tests, the manifest, every prompt and response under model-calls/
  execute/<package>/       verdicts per test on head, re-run and base; coverage; mutation/; relevance.json
  triage/<package>/        every verdict and finding classified, with confidence and a route; accounting.json, every reached behaviour in one state
  packet/<package>/        packet.md, the tests as a patch, the draft VEX, pull-request.md (what the test pull request carries)
  attest/<package>/        the provenance record per test, the signed statement, the UDLM records
  assess/                  the run against the blueprint's thirteen goals
  propose/<package>/       the test pull request the harness opened, and what it put on the branch
  feedback/<package>/      what a person accepted, edited or rejected; the realized records; a signed statement

One story, three layers

A run produces a lot of files. They are there for a reason, and the reason is written down, but a reader should never need them to understand what happened. Every run tells one story in three layers, each written by the harness from its own records so they cannot disagree.

1. The story. README.md at the top of the run. It says who did the work and who decided, what was tested, why it earned the budget it got, where it ran and when. It gives the outcome in the packet's plain terms and the decisions the harness made: what it cut, what it accepted, what it escalated. It lists the actions it took and the ones it did not take by design, states each UDLM record as one plain sentence, and names every file in the folder with the reader it is for. The site shows the same story at the top of each run page. Start here whoever you are. 2. The pull request. Tests reach a repository only through a pull request that a person merges. packet/<package>/pull-request.md is the text and the file list that pull request carries, written at packet time whether or not the run went on to open one, so a reader sees exactly what a reviewer would be asked to accept. packet.md beside it is the reviewer's brief: the verdict per test, the findings, the draft VEX statements. 3. The evidence. Everything else is proof and replay: the sealed sandbox runs, every prompt and response, the signed statements, the sealed records. Each of those files carries a label in run-index.json saying who it is for and why it exists; the appendix at the end of this document is that same table.

run-index.json at the top of every run says which stages wrote, which packages the run holds, the headline numbers, and where each file is, so a reader or a renderer never has to know this layout by heart. The site renders one page per run from it: what each stage read, did and wrote, the results per package, the assessment, and every artifact linked, at croadfeldt.github.io/ai-test-harness/runs.

Three rules hold everywhere. The harness never merges, publishes or deletes; every file is a proposal. A test is "proven" only when it fails on the vulnerable version and passes on the fixed one, both runs kept. "Unproven" is stated, never hidden behind a green test.

The dashboard, and the runs store behind it

One run tells one story. The dashboard shows every run, over time, for three readers. Leadership sees what went through, what was proven, and what a person acted on. Engineering sees which model and prompt set is fit for which job. Pipeline owners see what the harness found about the CI it ran in. Filters by repository, language, model and prompt set. It is rendered from the same run indexes and evaluation rows the stories are written from, so it cannot disagree with them.

A run reaches it through the runs store. harness publish puts four small files per run, the index, the story, the evaluation row and the pipeline findings, into a directory or a git repository named by HARNESS_RUNS_STORE. The site build reads that store beside the example runs. harness run publishes by itself when a store is configured; the CI examples do the same. The store is the harness's ledger: files are added, never rewritten, and nothing under review lives there.

If you are reviewing the change

Open packet/<package>/packet.md. Read the first paragraph: the verdict in plain words, what the change did, what was proven, what was not, what to do. Then read "What was accounted for, and what was not" in the run's README.md. Every behaviour the change reaches is validated, handed to you for your environment, or a stated gap with its reason. The pull request text repeats the gaps as "Not covered". Then:

The tests themselves are packet/<package>/tests.patch, in the layout of the overlay repository, or under tests/ for the application's own code. In the pipeline's loop the same content arrives as a pull request on the overlay repository with this packet as its text; review it like any contributor's. Budget about a minute per package.

If you are Product Security

Open packet/<package>/vex.openvex.json. Each statement's status is a draft: fixed only when a fix-pinning test proved it, affected when the advisory is open at head and the application reaches the package, under_investigation otherwise; the basis is in the statement. For any fixed, the proof is the test named in the basis and its two runs, execute/<package>/old/ and new/ (or fixed-candidate/), logs included. A blocked on both versions verdict in execute/<package>/results.json is a reproducer for a possible defect in the package itself, worth a look on its own.

If you are Supply Chain Security or an auditor

Open attest/<package>/. statement.dsse.json is the in-toto test-result statement signed as a DSSE envelope; signer.pub.pem verifies it, and attest.json says which key (a development key today, Trusted Artifact Signer in Konflux). Its subjects are the patch, the provenance record and each evidence record's head, so a signature check binds all three. MANIFEST.json is the provenance record per test: the model and endpoint by label and digest, the prompt and response digests, the sandbox image digest, the verdicts. udlm/ holds the same facts as sealed records: the vulnerability, the package, the run as a job, one test-evidence record per accepted test, the draft VEX; udlm/index.json says whether they sealed and validated. After a person accepted the pull request, feedback/<package>/ adds the requested and realized records for what landed and a signed acceptance statement naming the merge commit.

To verify a statement on this machine:

cd harness && .venv/bin/python -c "from pathlib import Path; from harness.stages.attest import verify; \
print(verify(Path('<run>/attest/<package>/statement.dsse.json'), Path('<run>/attest/<package>/signer.pub.pem')))"

If you lead the engineering team

Open assess/assess.md: thirteen goals from the blueprint, each with the measurement, the verdict and the file it was measured from. Whether a run can be trusted is answered there, not in the packet. Then execute/summary.json for the counts, execute/<package>/mutation/mutation.json for how strong the accepted tests are (score, and which test killed which mutant), and execute/<package>/relevance.json for what the change makes obsolete or redundant among the tests you already have. The roll-up across every run is 11-evidence-rollup.md.

If you are the developer who ran it

You are lifecycle A (blueprint 5.0). Read the packet, keep the tests that look right from tests.patch, drop or edit the rest, and run again if you changed the code; your pull request carries the tests and the records. Two things to know: generate/<package>/manifest.json lists what was discarded and why, with the last problem the model was sent back for; and generate/<package>/scratch/ holds every attempt that ran in the sandbox, so a cut test can be recovered.

If you want to know why the harness did something

generate/<package>/model-calls/ has every prompt and every response, numbered, with the call's settings and token counts; for the tool-using variant, every tool call and result as the model saw it. selfcheck/selfcheck.json is stage 0: which register checks ran and what the probes found. run.json names the tools and versions. Nothing about the machine, the network or the account that ran it is recorded: endpoints appear as a label and a digest, repositories by name.

Appendix: every file, its reader, its reason

Four machine files a run writes are not committed with the example runs, because a run recreates them and they carry no fact the committed files lack. They are the API surfaces at head and base, the resolver's two dependency graphs, and the raw coverage profile; its parsed form, coverage.json, is kept. The catalogue below still names them, and a run on your machine has them.

Six audiences: everyone; reviewers (the developer or team that owns the change); Product Security; Supply Chain Security and auditors; the person who ran the harness; and tools, for files kept as proof and replay that nobody is expected to read. Paths are shown without the package directory; in a run they sit under <stage>/<package>/.

Generated from harness/src/harness/runindex.py by tools/catalogue.py; the run index carries the same label on every file it lists, and the story at the top of each run groups the files by it.

Everyone

Reviewers (the developer or team that owns the change)

Product Security

Supply Chain Security and auditors

The person who ran the harness

Tools (kept for replay and proof; not meant to be read)

The evidence so far

For: anyone deciding whether this is worth their time. Every number below is read from a run's records under examples/ by tools/rollup.py; nothing here is typed by hand, and a run that lacks a stage shows a dash.

In plain terms. 17 runs with executed tests, 19 package rows, 193 generated tests run in a sealed sandbox, 12 tests proving a vulnerability closed or open by failing on the vulnerable version and passing on the fixed one, 65 tests accepted by triage as candidates, 15 signed attestations. "Proven" counts tests, so one vulnerability may carry two. Mutation is the share of sampled mutants the accepted tests killed, target 0.6. "Accepted by review" is what a person kept after the harness proposed, from stage 7. Time from trigger to packet is on each run's own assessment (goal G9), because several of these runs were re-attested days later and the records would say so.

How to read any one of these runs, file by file, is in 10-reading-the-artifacts.md. The story behind each run is on its example page: frc-scheduler-server (Python) and control-plane (Go).

Prompt and model evaluation

For: anyone deciding which model or prompt set to run, and anyone asking whether the harness's results can be trusted. Generated from the example runs by tools/prompt-eval.py; the same table for any set of runs comes from harness evaluate.

Fitness for purpose is measured, never assumed. Each line is one prompt set with one model on one package, summed over the runs that used them. Ground truth in order of strength: a reviewer's decision on the test pull request, the sandbox's verdicts (a fix proven means the test fails on the vulnerable version and passes on the fixed one), triage's acceptance. "Validated / reached" is the accounting: of the behaviours the application reaches, how many a kept test ran on both sides with a mutant killed inside them (principle 15); it reads lower than accepted, and should. "For review" counts tests whose behaviour differed between the two versions, which a person must judge. Covered lines and the mutation score say how much of the package the tests exercise and how firmly. Cost is model calls and minutes of model time.

Generated 2026-10-15T19:35:47+00:00 from 19 package run(s).

The runs behind each line

A prompt set or a model that does not reach the outcome a package needs (a proven fix for an advisory, an accepted characterization otherwise) is not fit for that purpose, whatever it scores elsewhere. The harness records the set's name and digest in every manifest, so any line here can be traced to the exact words and the exact model that produced it.

Running it on real repositories

For: the engineer wiring the harness into a repository's pipeline, and the reviewer who wants to know what will run when. One command, any git repository, any ref; then the triggers that call it from GitHub Actions, GitLab CI, Tekton, or a schedule. The human gate is unchanged: the harness only proposes, a person merges.

One command, one change

harness run --repo https://github.com/org/app.git --base main --head refs/pull/123/head --workdir out/pr-123

That runs every stage in order on the difference between the pull request and its base: intake, analysis, generation, execution, mutation, relevance, triage, packet, attestation, assessment. The work directory ends with the story at its top (README.md), the review packet and the pull request text under packet/, and the signed records under attest/. Add --propose to open the test pull request on the overlay repository afterwards; the harness pushes one branch and never merges.

--repo takes a path on the machine or a git URL (https, ssh, file). A URL is cloned once under the run's cache and fetched on later calls; the harness checks its own clone out at the change and never touches a checkout it did not make. --head and --base take anything git can name on that repository:

Pull request refs are fetched by name, so the harness never needs the branch to exist locally, and the repository name in every record comes from the URL, not from the directory it was cloned into.

--target first-party tests the repository's own code instead of its dependencies; all does both. --select narrows to named packages; without it, every package the change touched or that carries an advisory is in scope, at the budget its risk score earned.

What every trigger needs

• The harness with its sandbox: pip install from this repository, plus podman on the runner, or the container image with the pod sandbox target inside a cluster. Generated tests run before anyone has read them, so a runner without a sealed sandbox is not an option.
• A model endpoint: HARNESS_MODEL_BASE_URL, HARNESS_MODEL, and a key if the endpoint wants one, from the platform's secret store. The endpoint's address is recorded only as a label and a digest.
• Somewhere for the output: the work directory as a build artifact, and, in the pipeline's loop, credentials for the overlay repository so --propose can open the test pull request.
• Time: budget an hour or more for a package with advisories on a served model; the run is validation, not the suite's regression run, and it happens once per change.

GitHub Actions

deploy/github-actions/ai-test-harness.yml is a complete workflow. It runs on three triggers:

• pull_request: base is the pull request's base branch, head is the pull request's head ref. The packet's verdict in plain terms is posted as a comment on the pull request; the run directory is uploaded as an artifact. Nothing is merged.
• push to the default branch: base is the commit before the push, head is the pushed commit.
• schedule: a rescan of the default branch, so advisories published since the last change still produce tests.

workflow_dispatch lets a person run any two refs by hand. Secrets: the model endpoint and key, and a token with write access to the overlay repository if the workflow proposes.

GitLab CI

deploy/gitlab-ci/ai-test-harness.gitlab-ci.yml does the same with GitLab's variables: on a merge request, CI_MERGE_REQUEST_TARGET_BRANCH_NAME is the base and refs/merge-requests/<iid>/head the head; on a push to the default branch, CI_COMMIT_BEFORE_SHA and CI_COMMIT_SHA; on a pipeline schedule, a rescan. The run directory is a job artifact; the verdict goes to the merge request as a note through the API.

Tekton, on OpenShift

The pipeline in deploy/tekton/ already takes repo-url, base and head as parameters. The triggers under deploy/tekton/triggers/ connect a GitHub webhook to it. An EventListener receives the event, a TriggerBinding maps a pull request's base and head ref (or a push's before and after commits) onto the parameters, and a TriggerTemplate creates the PipelineRun. The same pipeline runs on a schedule as a CronJob that creates a PipelineRun with head set to the default branch and no base.

What the harness says about the pipeline it runs in

Fitting into a repository's CI is the default; fitting in silently is not. Every run reads the CI definitions at the reviewed commit and states, as findings routed to the pipeline's owners, where that CI is provably short of what validating a test needs. Two questions today:

• Can untrusted code reach secrets or the network while tests run? A pull request checked out under the base repository's secrets, write permissions on a pull request workflow, actions resolved by a mutable tag, a download piped to a shell, a test job with repository secrets in its environment, a self-hosted runner for pull requests, a floating image tag, a privileged step. Each finding names the file, what was found, and the change that corrects it. A repository with no CI at all is a finding too.
• Does the existing suite reach what changed? For each package in scope the harness counts the repository's own test files that reference it and those that reach the symbols the change altered. Zero, where the application reaches the package in production, is a coverage gap with the call sites to test.

They appear in the packet under "The pipeline this change runs through", in the story's decisions, and in triage/pipeline.json. The harness changes nothing in the pipeline; the owners decide. The harness reads its own workflows the same way, and reports that its actions are pinned by tag.

Opening the test pull request from CI

The pipeline's loop ends with the test pull request, and every engine can open it. What a run needs:

• Where the tests go. Dependency tests go to the test overlay repository: HARNESS_OVERLAY_REPO (a checkout) or the HARNESS_OVERLAY_URL the examples clone. First-party tests go to the repository under test, as a request against the change's own branch, so they land in the developer's pull request. When the change was a commit or a pull request ref, the request targets the default branch. - A token that can push a branch and open a request there: GH_TOKEN for GitHub, GITLAB_TOKEN for GitLab, or HARNESS_FORGE_TOKEN in the examples, which set both. The harness uses gh or glab when present and the forge's API when not; any other forge gets the pushed branch and a message to open the request by hand. - An identity: HARNESS_PROPOSE_AUTHOR and HARNESS_PROPOSE_EMAIL, the bot the requests are authored under, and optionally a signing key ([propose].sign).

The request's text is the packet's plain-terms verdict, the numbers, the file list, and the run's story folded below it, so a reviewer has the whole account without leaving the request. The harness pushes that one branch and nothing else, and it never merges.

Every example does the same five things. It installs the harness, runs every stage on the two refs the event implies, keeps the run directory as the build's artifact, posts the plain-terms verdict where the change is reviewed, and opens the test pull request. With HARNESS_RUNS_STORE set they do a sixth: publish the run's index and story to the runs store, so it appears on the dashboard. The generic script is those five steps in shell; the others are the same steps in each engine's own syntax.

From a developer's assistant

skills/ai-test-harness/ is an Agent Skill: the instructions a coding agent such as Claude Code loads to run the harness on the branch in front of the developer, read the story, and propose the tests. The rules travel with it: sealed sandbox only, nothing merged, unproven stated as unproven. Copy it into .claude/skills/ and lifecycle A is one instruction away.

Anything else

Any system that can run a command with two refs can trigger a run: a cron entry, a webhook receiver, a person. harness run is the whole pipeline; the stage commands remain for running it a step at a time. Every trigger produces the same artifacts and the same story, so a reviewer reads a run the same way whatever started it.

Implementation specification

For: the engineer implementing or evaluating an AI Test Harness. One page. What an implementation must do, the three contracts it must honor, and how it is judged. The reasons are in the blueprint and its reference; this page states only what is required. MUST is required; SHOULD is expected unless a written reason says otherwise. Every line names the check or test that proves it in this implementation, or says there is none yet.

Requirements

Contracts

• Adapter toolkit. One module per ecosystem exposing the same names: SYSTEM, CATEGORY_TASK, prompt_preamble, extract_code, compile_check, test_names, drop_tests, cut_at_errors (compiled languages), imports_ok, weak_assertions, prefetch, run_tests, parse_results, coverage_for, mutation_sites, apply_mutation, write_overlay, mutant_status, fixed_candidate, symbol_refs. The stages call nothing else.
• Sealed sandbox. Input: an offline environment, a tests directory, an optional overlay for a mutant. Output: junit.xml, coverage.json with executed lines per file, the run script, stdout and stderr, and a summary with the image digest, timeout and isolation facts. Same shape for every target and language.
• Records. run-index.json (format ai-test-harness/run-index/v1), README.md the story, packet/<pkg>/{packet.md,tests.patch,vex.openvex.json,pull-request.md}, attest/<pkg>/{MANIFEST.json,statement.json,statement.dsse.json,udlm/}, feedback/<pkg>/{acceptance.json,decisions.jsonl,udlm/}. Schemas under blueprint/.

How an implementation is judged

1. The register passes: every entry has a check, and stage 0 runs them all before every run.
2. Every run is assessed against the blueprint's thirteen goals, with the measurement and the file it came from.
3. The reference jobs, rerun and compared with the baseline in document 12:

A change that lowers a baseline line is a regression until the table says otherwise; a model or set that beats one becomes the default for that job.

Generation failure register: detect, correct, verify

Status: living document, moved here unchanged from the blueprint (v0.18 section 17) on 2026-10-01. Owner: Chris Roadfeldt. For: anyone adding a register entry, reading a stage 0 record, or checking that a known failure mode is covered. The blueprint states the principle; this document is the register itself. The code that runs it is harness/src/harness/selfcheck.py.

The generator will fail in ways nobody predicted. That is not the problem. The problem would be failing the same way twice, or shipping a test that looks green because a failure went unnoticed. This document is the contract that turns every observed failure into a permanent, verified control.

The contract

Every entry in the register has six parts, and an entry is not complete until all six exist:

Two rules follow. Fail closed: a run whose self-checks do not all pass produces no evidence. Register first: when a new failure mode appears, its entry and self-check are written before the correction, so the self-check fails on the old code and passes on the new. That is how the harness applies its own test-first discipline to itself.

Where a failure cannot be corrected automatically, the correction is an honest verdict in the run's output and an escalation to a person, never a quieter failure. "No valid trigger produced for this advisory" is a correct result. A green test that proves nothing is not.

The register

Every entry below was observed in a real run on examples/frc-scheduler-server and is reproduced by a self-check in harness/src/harness/selfcheck.py. Ids are stable; entries are never deleted.

How the register grows

1. A run produces a failure the register does not name. The evidence is in the run's artifacts; the run is kept as it is.
2. The entry is written with all six parts. The self-check is committed first and fails.
3. The correction lands. The self-check passes. The next run's stage 0 record shows the new id.
4. The example that surfaced the failure is rerun with only that correction changed, and the before and after are kept side by side. Runs 1 through 5 on python-jose are the first instances; run 4 alone added GF-013 to GF-015.

The register is reviewed with every prompt or model change (workflow 12), because a new model fails in new ways, and the quarterly report (workflow 15) lists entries added in the quarter.

Blueprint reference: the long form

Status: v0.18 of the blueprint, kept as the reference behind the three-page blueprint. Moved here on 2026-10-01 minus the failure register, now document 15, and minus three sections that other documents already carried: section 4 points at documents 04 and 05, section 7 at document 05, section 13 at document 07. Owner: Chris Roadfeldt.

How to read this: it is a reference, not a read-through. Fifteen thousand words. Start from the three-page blueprint and come here when it cites a section; the section numbers are stable because the code's docstrings and the other documents cite them.

For: the reader who has decided the blueprint is worth a closer look and wants the reasoning, the stage-by-stage detail, the lifecycle and provenance design, the metrics, risks, rollout, open questions and the worked example. Section numbers are the ones the code's docstrings and the other documents cite ("blueprint section 8.2"). New material goes to the three-page blueprint first and here second.

Changes in v0.18: the pipeline's loop is complete on every common engine. The test pull request opens on GitHub or GitLab through their tools or their APIs (stage 6, lifecycle B). First-party tests go to the repository under test against the change's own branch, and the request carries the run's story. The examples under deploy/ wire GitHub Actions, GitLab CI, Jenkins, Azure Pipelines and Tekton, with one script for any other engine. An Agent Skill makes lifecycle A one instruction away in a developer's coding agent.

Changes in v0.17: the pipeline is under test too (principle 14). The harness fits into the repository's own CI. When that CI is provably short of what validating a test needs, it says so with evidence and the change that corrects it, routed to the pipeline's owners. It never edits the pipeline. Two questions first. Can untrusted code reach secrets or the network while tests run? Stage 2 reads the CI definitions and stage 5 classifies the facts. Does the existing suite reach what changed? Stage 2 counts it and stage 5 states the gap. Goal 12 records that the questions were asked.

Changes in v0.16: the rules the first measured prompt comparison and the 235B runs forced. In stage 3 the compiler's line cuts a test before any repair, and a worse repair never erases an attempt that ran; in stage 4 mutants come only from lines the tests executed. Prompt sets ship with the image and are selected per run, the repository is matched by its name in every stage, and the register stands at twenty-eight. Principle 12 has its first worked example under stage 7. A one-page implementation specification, document 14, now states what an implementation must do and how it is judged.

Changes in v0.15: the harness takes a repository by address and any ref git can name on it, a pull request's head ref included, and runs every stage with one command (stage 1; document 13). A pull request trigger, a push, a schedule or a person can start a run on any real repository. The triggers for GitHub Actions, GitLab CI and Tekton are documented and shipped as examples.

Changes in v0.14: fitness for purpose is validated, never assumed (section 3, principle 13; stage 7). Every text the harness says to a model is a named, digested prompt set under source control, and every manifest records which set and which model produced a result. The same jobs run per set and per model and are scored by what the sandbox and the reviewers said. A set or a model becomes the default only by winning that table. The evaluator and the bench are the implementation.

Changes in v0.13: every run must explain itself to a general reader (stage 6). The run's story is an output of every run: who, what, why, where and when, with the outcome, the decisions and the actions. So is the text and file list of the test pull request, whether or not one is opened. Every other file the harness writes carries a named reader and a reason, so evidence stays but never obscures.

Changes in v0.12: the harness is one part of a larger pipeline and it only tests (section 2.4). A separate fix pipeline generates fixes; this harness hands it failing tests, reproducers, the upgrade path and draft VEX statements, and tests whatever comes back. Section 9 no longer lists proposed production fixes among what the harness generates.

Changes in v0.11: the repository's own code is a target in its own right (stage 2, section 5.0). It gets one row at depth 0, facts from the tree at the reviewed commit, and tests proposed under the application's own tests directory. No advisory lookup runs for the application itself, so CVE-targeted work stays with the dependency rows unless a vulnerability is already known.

Changes in v0.10: the pipeline is one core with two lifecycles around it (section 5.0). A developer runs it inside their own work; a CI/CD pipeline runs it on every change and proposes a test pull request. Creation and execution can be separated as an organization sees fit, the human review gate is explicit, the harness's validation run and the suite's regression run are named apart, and tests always arrive through a pull request.

Changes in v0.9: the provenance record is a UDLM record (section 8.2). Test evidence, the draft VEX statement, the vulnerability, the package, and the run are records in the Unified Data Lifecycle Model, each with UDLM's own tamper-evident head. The signed in-toto statement is the projection of those records, and each evidence record's head is a subject of the statement. This is the shape UDLM adopted in its own registry (TestEvidence in three tiers, VexStatement) after the first real run produced them.

Changes in v0.8: added stage 0, self-verification, to the pipeline (section 5) and the generation failure register (document 15). Every way the generator has been observed to fail is recorded with the issue, why it matters, the cause, the automatic correction, and the self-check that proves the correction is in place on every run. The pipeline fails closed if a self-check fails. Stage 7 feeds new failure modes into the register. Two metrics added to section 10. Appendices renumbered.

Changes in v0.7: incorporated the orchestration and capability research (05-capability-map.md). The test-evidence attestation now uses the vetted in-toto test-result/v0.1 predicate. The tooling table names the agent runtimes, sandbox, gateway, observability, and eval choices, and Konflux deptriage is recorded as the in-org precedent.

Changes in v0.6: added the native-tooling guiding principle: generated tests conform to the ecosystem's existing testing paradigms and run with its native tooling so they drop into existing standard operating procedures and CI/CD pipelines without a harness-specific runner.

Changes in v0.5: added the chain-of-trust guiding principle and its consequences for the harness's own provenance (section 3, 8.5). Added section 16, future enhancements: verifying that change descriptions match the change, and detecting malicious or vulnerable changes across first, second, third, and transitive parties. Appendices renumbered.

Changes in v0.4: added SLSA v1.2 compliance targets for provenance and attestation (section 8.5). Added negative tests and CVE-targeted tests as generation categories with a VEX evidence path (section 5, stage 3), with matching metrics, phases, security rules, and open questions.

Changes in v0.3: added two guiding principles: generated tests are provenance and candidates for the standard suite, and the harness identifies tests made irrelevant by code changes. Expanded section 8 into a test lifecycle covering provenance, promotion, and retirement. Added obsolete and redundant triage classes, and lifecycle metrics, phases, and risks.

Changes in v0.2: added section 4 (build on existing work), replaced the generic tooling table with named projects from the landscape research, added benchmarks, added a licensing note, and tightened the open questions.

1. Purpose

Every change that enters our software supply chain arrives with uneven test coverage. First-party pull requests usually carry some tests. Third-party libraries carry whatever their upstream wrote. Transitive dependencies, pulled in several layers deep, are effectively untested from our point of view: we never wrote a test for them, we rarely read them, and we find out they broke something only when a product does.

This is my full blueprint for an AI Test Harness: a pipeline that uses AI agents to generate, execute, and validate unit tests, functional tests, and supporting harness code for every incoming unit of source code. That unit may be our own commit, a direct dependency, or a transitive dependency several levels removed. It is the most detailed document in this repository. Everything else summarizes it.

The harness is one part of a larger pipeline. It tests; it does not fix. Generating fixes belongs to a separate pipeline that consumes what this one produces (section 2.4). Keeping that line is a design choice, not a gap.

The goal is not "AI writes our tests." The goal is evidence: for each incoming change we want a machine-produced, human-reviewable, reproducible answer to three questions.

1. What does this code actually do? (characterization)
2. Did its behavior change from the version we had before? (differential)
3. Does it do anything it should not, and is it exposed to what is already known to be wrong with it? (negative tests and CVE-targeted tests)

2. Scope

2.1 In scope

2.2 Out of scope (for now)

• Binary-only artifacts with no source (handled by a separate binary analysis track).
• Performance and load testing (may be added in a later phase).
• Automatically merging AI-generated tests without human review (never in scope). An organization may let policy accept low-risk characterization tests without a reader, but the merge is still a pull request, and the harness never pushes to a protected branch.
• Fixing production code, automatically or as a proposal. A separate fix pipeline does that, fed by this harness's outputs; section 2.4 says why the line is drawn here.

2.3 Target ecosystems, in priority order

1. Go
2. Python
3. Java and Kotlin (Maven, Gradle)
4. Rust
5. JavaScript and TypeScript (npm)
6. C and C++ (CMake, Autotools, RPM spec driven builds)

Each ecosystem needs a small adapter (section 7, document 05). The core pipeline is ecosystem-agnostic.

2.4 Where the harness ends: the fix pipeline

This harness is the testing half of a larger pipeline. A second pipeline, the fix pipeline, generates changes to production code. The two are separate on purpose, and the boundary is a contract.

What this harness hands over. A failing test with both runs kept, which is the proof that a problem exists and the acceptance test for any fix. A reproducer for a defect the run found on the way. The upgrade path, when a vulnerability is fixed by a version the change did not bring: which versions have to move for the fixed one to resolve. A draft VEX statement per advisory. All of it in the packet, signed, with the provenance of every test.

What the fix pipeline hands back. A change to production code or to dependency versions, as a pull request. That change enters this harness like any other: the same stages, the same sandbox, the same differential run, and the failing test that motivated the fix is the first thing the run reports on. A fix is proven the way a vulnerability is, by a test that fails before and passes after.

Why the harness only tests. Four reasons, in order of weight.

1. Evidence must be independent of the thing it judges. If the pipeline that writes the fix also writes the test that proves it, the test tends to prove the fix rather than the requirement. Keeping generation of tests and generation of fixes in different pipelines, with different prompts, different budgets and different reviewers, is the cheapest independence there is.
2. The register works because the harness's failure modes are testing failure modes. Twenty-two entries, each with a check that runs before every run, describe ways a generated test can mislead. A fix pipeline fails in different ways, needs its own register, and mixing the two would dilute both.
3. Review stays tractable. A reviewer of a test pull request reads tests; a reviewer of a fix reads production code against a failing test. Separate pull requests, separate reviewers when the organization wants them, separate blast radius when one is wrong.
4. The trust model is narrower. This harness proposes tests, never merges, and never pushes to a protected branch; its bot identity needs no more than that. A pipeline that changes production code needs a different identity, different credentials and different policy, and should not inherit this one's.

The upgrade path is the one place this harness comes close to a fix, and it stays a description: which versions would have to move, recorded in the packet for the fix pipeline and the reviewer, never applied.

3. Guiding principles

Fifteen, numbered; the three-page blueprint lists the same fifteen in the same order.

1. Generated tests are hypotheses until executed. A test the AI wrote but the harness did not run against the real code has zero evidentiary value. Every generated test is compiled, run, and scored before a human ever sees it.

2. Tests must be strong, not just green. A passing test that would also pass against broken code is noise. Mutation testing and differential runs against the previous version are how we tell strong tests from weak ones.

3. Incoming code is untrusted input. Third-party source, README files, docstrings, and commit messages can contain prompt-injection payloads. The agent treats all of it as data. Generated tests run in a sandbox with no network and no credentials.

4. Humans own the merge. The harness produces review-ready artifacts. It does not merge, publish, or change policy.

5. Everything is reproducible. Each run records the model, prompts, tool versions, dependency graph, and container image digest. Any run can be replayed.

6. Cost is a first-class metric. Transitive graphs are large. The harness must prioritize by risk and stop when the marginal test is not worth the compute.

7. Reuse before building. The landscape research found that every stage of this pipeline except the assembly already has a maintained open source component. We build the assembly and the generation and triage agents. We do not rebuild SBOMs, mutation engines, or reachability analysis.

8. Native first. Generated tests look like the tests the team already writes and run with the commands the team already runs. Idiomatic framework, idiomatic layout, idiomatic naming, standard coverage and reporting formats. go test, pytest, mvn test, cargo test, npm test, ctest run them with no harness present. A generated test that needs the harness to execute has failed this principle. The point is adoption: tests that fit existing standard operating procedures and CI/CD pipelines get run, kept, and trusted. Tests that need special handling get ignored.

9. Generated tests are provenance. A test the harness produced is a first-class artifact of the build, not a side effect. It carries a signed record of what it was generated from, by which model and prompt, against which version of the code, and who approved it. That record is attached to the build's attestation so that "this package version was characterized by these tests" is a verifiable claim. And they are candidates for the standard suite. The overlay is a proving ground, not a final destination. A generated test that reviewers approve, that keeps killing mutants, and that survives version bumps becomes a standard test scenario for that package or product, on equal footing with human-written tests, and future harness runs start from it.

10. The harness extends the chain of trust. It never breaks it or launders it. Every input the harness consumes is attested upstream: the source revision, the Hermeto SBOM, the build's SLSA provenance. Every claim the harness makes references the attestation it was derived from and is itself attested. Every consumer downstream, Conforma, Trustify, a release pipeline, a customer reading a VEX statement, can walk the chain back. Where the harness cannot verify something, it says "unverified," it does not turn an unverified input into a verified-looking output. And the harness is a link in the chain itself: its code, prompts, container images, and models are built and attested the same way as anything it tests.

11. The harness tests itself before it tests anything else. Every failure mode the generator has ever shown is a register entry with a detector, an automatic correction, and a self-check that proves the detector and correction still work. The self-checks run as stage 0 of every pipeline run and fail closed. A failure mode without a self-check is an open defect against the harness, not a known limitation. Document 15.

12. The harness retires tests as well as writing them. Code changes make tests irrelevant: the symbol they exercised is gone, the behavior they asserted was intentionally changed, or another test now covers the same paths. The harness identifies those tests, explains why, and proposes their retirement. It never deletes a test on its own.

13. Fitness for purpose is validated, never assumed. A model and a prompt are inputs like any other, and their quality is a claim until it is measured. The harness therefore keeps every prompt as a named, digested set under source control and records the set and the model in every manifest. It runs the same jobs under each candidate and scores them by ground truth in order of strength. Strongest is what a reviewer accepted on the test pull request; then what the sandbox proved, a fix-pinning test that fails on the vulnerable version and passes on the fixed one; then what triage accepted. A model or a set that does not reach the outcome a job needs is not fit for that job, whatever it scores elsewhere, and does not become the default. The failure register (document 15) keeps the harness honest about its own defects; the evaluation table keeps it honest about the model's and the prompt's. Stage 0 proves the harness is fit to run; stage 7's evaluator proves the model and the prompt are fit to produce.

14. The pipeline is under test too. The harness fits into whatever CI a repository already runs, but it does not inherit that pipeline's gaps into its own evidence. Where the pipeline is provably short of what validating a test needs, it says so as a finding with evidence and the change that would correct it, routed to the pipeline's owners, and it changes nothing itself. Five questions are the whole of it, each answerable from facts. Does anything reach the base branch without tests? Can untrusted code reach secrets or the network while tests run? Does the existing suite reach what changed? Are failures being masked? Are the suite's results tamper-evident? The second and third are implemented; the others follow. Fitting in is the default. Fitting in silently is not.

15. Every reached behaviour is accounted for. The harness's output is not a list of tests; it is an account of the behaviours the change reaches. Each one ends in exactly one of three states. Validated: a kept test ran its lines on both sides of the change, ran consistently, and a mutant placed inside those lines was killed. For your environment: a real test that needs something only the team has, handed over with its declaration. Stated gap: the harness could not produce a test it trusts, or could not prove the one it has, and the record says why in one reason. Nothing is silent, and nothing is implied that was not earned: a test that executed a function without a mutant to prove it checks anything is a gap named strength-not-sampled, which is the harness's own failure, not the test's. The counts sum to the number of behaviours; a behaviour without a state fails the run's assessment (G13). This is what the business case promises the gate, and the stage 5 accounting is where it is kept.

4. Build on existing work

Nothing in the open source world, from Red Hat, IBM, or elsewhere, does the whole job; the pieces do most of the individual stages. The inventory and the reuse decisions are in document 04, section 6 (reuse directly, learn from, benchmark against, avoid, licensing), and the choice per capability with its fallback and reason is the decision table in document 05. This section keeps only what the harness builds itself.

4.5 What we build ourselves

1. The assembly: the work-list model, stage orchestration, and the adapter interface.
2. The generation agents for unit, functional, fuzz, and harness code.
3. The triage agent and its classification schema.
4. Cross-ecosystem differential testing of a consumer against a candidate dependency version. No maintained open source tool does this outside distro infrastructure.
5. Reachability for test targeting rather than for vulnerability matching, layered on CLDK, govulncheck, and dep-scan.
6. The test overlay repository format, its provenance manifest, and the promotion path into the standard suite (section 8).
7. The test relevance engine that maps every test to the symbols and paths it exercises and flags tests that code changes have made obsolete or redundant (section 8.4).
8. CVE-targeted test generation that turns advisory data into exposure tests and fix-pinning regression tests, and emits draft VEX evidence (section 5, stage 3).
9. Rust fuzz harness generation, since OSS-Fuzz-gen does not cover it.

5. Pipeline architecture

 ┌──────────┐   ┌──────────┐   ┌────────────┐   ┌───────────┐   ┌──────────┐   ┌────────┐
 │ 1 Intake │ → │ 2 Analyze│ → │ 3 Generate │ → │ 4 Execute │ → │ 5 Triage │ → │ 6 Review│
 └──────────┘   └──────────┘   └────────────┘   └───────────┘   └──────────┘   └────────┘
       ↑                                                                             │
       └──────────────────────── 7 Feedback / learning ──────────────────────────────┘

5.0 One core, two lifecycles

The seven stages are one core: gather facts, generate, validate in a sealed sandbox on both versions, triage, attest. Who runs that core, and where the accepted tests go afterwards, differ. There are at least two lifecycles, and an organization may separate creation from execution as it sees fit.

Two executions, named apart. The harness's run is validation execution: both versions, sealed, once, at acceptance time, producing the fix-pinning proof, the coverage, the mutation score, and the provenance record. The suite's run is regression execution: head only, every change, by the CI that already runs the suite. Validation is stage 4; regression is not a harness stage at all. An accepted test is simply a test, and it runs where tests run.

The review gate is a stage with inputs and outputs, not a principle. Its inputs are the packet, the differential verdicts, the provenance records, and the draft VEX statements. Its outputs are tests merged by pull request, a VEX decision for Product Security, and the reviewer's decisions captured for stage 7. In lifecycle A the gate is code review; in lifecycle B it is the test pull request. The organization sets the gate's strictness, up to and including policy that accepts low-risk characterization tests without a reader, but the merge is always a pull request, and the harness never pushes to a protected branch.

Trust differs by lifecycle. In A the developer is the author and the harness is a tool they used; the change is reviewed as their work. In B the harness is a contributor, and its pull requests are treated the way any contributor's are: a bot identity, signed commits, provenance attached, and the source-level SLSA claims of section 8.5 on the tests it proposes. That is what makes lifecycle B safe to leave unattended.

One rule holds in both, and it is the one most likely to be skipped in lifecycle A. Generated tests run before anyone has read them, so they run in the sealed sandbox of section 11, never in a plain test runner on a developer's machine. The developer flow uses the same isolation the pipeline does.

Where the implementation stands: the command line and the Tekton pipeline in harness/ are the core as lifecycle A runs it, and they produce the packet lifecycle B needs. harness propose is lifecycle B's last step. The accepted tests, the packet, the provenance record, the signed statement and the UDLM records go on a harness/ branch of the overlay repository under the harness's own identity, and a pull request is opened. That branch is the only ref the step pushes; nothing in it can merge. harness feedback closes the loop after a person has decided (stage 7). The same stages take the repository's own code as the target (analyze --target first-party). It gets one work-list row at depth 0, facts from the tree at the reviewed commit, the tree on the sandbox's import path, and the tests proposed under the application's own tests/. No advisories are looked up for the application itself, so CVE-targeted work stays with the dependency rows unless a vulnerability is already known.

Stage 0: Self-verification

Before the harness touches the incoming change, it proves it is fit to run:

• Every entry in the generation failure register (document 15) has a self-check, and every self-check passes. A self-check exercises the detector and the correction on a fixture that reproduces the original failure, in the same process and configuration the run will use.
• The sandbox's isolation claims hold: no network, no inherited environment, read-only root, offline install from the wheelhouse. Proven by running a probe file inside the real sandbox.
• The model endpoint answers, reports its model id, and honors the reasoning and sampling settings the run will record in its provenance.
• The adapter's tools are present at the versions the run will record.

The result is a selfcheck record listing every check, its register id, and pass or fail. It is written to the work directory, referenced by every manifest the run produces, and attested with the rest of the run's evidence. Any failure stops the pipeline before stage 1. A run whose self-check record is missing or failing produces no evidence anyone should trust, and Conforma policy treats it that way.

Stage 1: Intake and inventory

The input is a repository by path or by address and two refs: the change's head and the last known-good base; a base omitted means a rescan of one ref. Any ref git can name on the repository is accepted, a pull request's own ref included, so the trigger decides what is compared. A pull request is compared against its base branch, a push against what it replaced, a release against the last one, a branch alone on a schedule. A repository given by address is cloned once under the run and never touched where a person is working.

• Detect the incoming change: a build platform's snapshot (Konflux's, in the example), PR webhook, lockfile diff, SBOM diff, or scheduled scan.
• Resolve the full dependency graph with Hermeto, which also yields the SBOM. Fall back to the native resolver (go mod graph, pipdeptree, mvn dependency:tree, cargo metadata, npm ls --all) where Hermeto's support is experimental.
• Diff the SBOM against the last known-good SBOM via Mobster and Trustify.
• Run the pre-flight gates: OpenSSF Malicious Packages lookup, GuardDog, Capslock for Go.
• Emit a work list of (package, old_version, new_version, depth, reachable, preflight) tuples. Depth 0 is our own code. reachable comes from call-graph analysis: is any symbol in this package invoked from our code, directly or transitively?

Stage 2: Analysis

For each item on the work list, the agent collects context the test generator will need:

• Public API surface, extracted with CLDK, the language's own tooling, or the API diff tool for the ecosystem. Never by asking the model to guess.
• The source diff between old and new versions (OSSGadget oss-diff, or git) for updates, or the full source for new packages.
• The API contract diff from japicmp, cargo-semver-checks, griffe, gorelease, or API Extractor.
• Existing upstream tests, so we do not duplicate them and can reuse their fixtures.
• Changelog, release notes, and documentation, treated as untrusted text.
• Static analysis output: linters, gosec, bandit, semgrep, build warnings.
• Known vulnerabilities for the old and new versions from Trustify Dependency Analytics, OSV, and govulncheck, including the affected symbols or functions where the advisory names them, the fix commit where one is linked, and any public reproducer. Advisory text and reproducers are untrusted data like everything else from outside.
• A risk score (section 6.3) that decides how much generation budget this item gets.

For the repository's own code, the depth 0 row, the same facts come from the tree at the reviewed commit. They are its packages, their API surface, what the change added or altered, the dependencies its tests may import, and the tests it already has. No advisory is looked up for the application itself unless it is a published package with a known one. The harness does not go looking for vulnerabilities in first-party code; it tests the code as written and leaves CVE-targeted work to the dependency rows, where the vulnerabilities are already known.

Static analysis, as implemented. The design above names the tools an ecosystem can bring; the implementation runs these, and records which ran and which did not in every fact bundle:

Parameter edges are inferred from the declared type, not read from source. Python and Go carry no ranges (only Annotated[int, Ge(0)] does, and few codebases use it), so the worklist says "try 0, -1, a very large value" for an int and labels it so. Implicit paths a parser misses (an exception raised by a callee, a with block that raises) are not in the list; a control-flow pass is the next step there.

Not yet wired, and named here so no one assumes it. Call-graph reachability (CodeLLM-Devkit or the language's own tools) in place of "the application references this package". The ecosystems' contract diff tools (japicmp, gorelease, griffe) in place of the harness's own extractors. And semgrep. When a tool is absent the fact bundle says so (status: not_run), and the prompt carries no finding it did not have.

The pipeline, as facts (principle 14). Stage 2 also reads the repository's own CI definitions at the reviewed commit, GitHub Actions, GitLab CI or Tekton, and records what they do with untrusted code while tests run. The facts it looks for are a pull request checked out under the base repository's secrets, write permissions on a pull request workflow, actions by mutable tag, and a download piped to a shell. Also a test job with secrets in its environment, a floating image tag, and a privileged step. And for every package in scope it counts what the existing suite reaches: test files that reference the package, and test files that reach the symbols this change altered. Facts only; stage 5 turns them into findings.

Stage 3: Generation

The agent produces artifacts in four categories. Each is a separate sub-task with its own prompt and acceptance criteria. The loop follows ASTER: facts in, tests out, compile, run, repair, fill gaps.

Generation rules:

• Use the ecosystem's idiomatic framework (testing plus testify, pytest, JUnit 5, cargo test, Jest or Vitest, GoogleTest or Catch2). No custom frameworks, no harness-specific assertions, no runner the team does not already have. Tests land in the directory and naming convention the repository already uses, produce coverage in the format its CI already consumes (Cobertura, JaCoCo, lcov, JUnit XML), and are selectable with the tags or markers the team already uses so they slot into existing pipeline stages without a new job.
• For dependency updates, generate tests against the old version first, confirm they pass, then run them against the new version. Failures on the new version are behavior changes, not test bugs.
• For code we call, prioritize the call sites: the agent reads how we use the package and writes functional tests that mirror those usage patterns.
• Apply the assured-improvement filter: a generated test is kept only if it compiles, passes on the baseline, and raises coverage or kills a mutant that existing tests did not.
• Judge tests, not files, and attempts, not only the last one. Where the language compiles a file as one unit, the compiler's own line numbers cut the tests it names and the rest run before any repair goes back to the model; an unused import is pruned, not sent back (GF-024, GF-027). A repair that collects nothing never erases an earlier attempt that ran: that attempt is kept and its failing tests cut as usual (GF-027).
• The agent may propose production code changes only as separate, clearly labeled patches (for example, "suggested fix for a nil dereference found by fuzzing"). These never land automatically.
• Negative tests are generated for every function that accepts external input, at every depth where generation runs, and are never skipped for budget reasons: they are cheap and they are the tests most often missing from upstream suites.

CVE-targeted tests and VEX evidence

Every vulnerability that Trustify Dependency Analytics, OSV, or govulncheck reports against a package version in the graph gets a targeted sub-task, regardless of depth or risk score:

1. Locate. Identify the vulnerable symbols from the advisory, the fix commit diff, or the OSV affected-function data. If none is available, the agent proposes candidates from the diff between the last vulnerable and first fixed version and marks them as inferred. 2. Exposure test. Using the call graph, find whether our code reaches the vulnerable symbol. If it does, generate a test at our call site that drives the known trigger through our code path in the sandbox. The result is direct evidence of whether we are affected. 3. Fix-pinning test. Generate a test that fails on the vulnerable version and passes on the fixed version, the same fail-before, pass-after criterion used by TDD-Bench-Verified. This test is promoted to the standard suite as soon as it is approved, so the vulnerability cannot return through a downgrade, a fork, or a vendored copy. 4. Draft VEX. The outcome is written as a draft VEX statement in CSAF form, linked to the test provenance record. It says affected with the exposure test as proof, not_affected with a justification such as vulnerable_code_not_in_execute_path and the reachability evidence, or fixed with the fix-pinning test. Product Security confirms or rejects the draft. The harness never publishes VEX on its own. Confirmed statements flow into Trustify, which already ingests VEX.

Rules specific to this category: exploit-style inputs run only inside the sandbox. Tests for a vulnerability that is not yet public or not yet fixed are stored with restricted visibility in the overlay until disclosure, following the normal Product Security embargo process. Public reproducers are read as untrusted data and rewritten by the agent, never copied in.

Mutation feedback, as implemented (MuTAP's idea). Before a file that passes its baseline run is believed, the harness breaks the package on purpose in a few places the file's tests executed, inside the reached behaviours' closures first. It runs the file against each breakage in the sealed sandbox. A breakage every test lets through is a line the tests executed without checking, and it goes back to the model once, as a repair round, with the diff and the instruction to add the check that line deserves. The file is then probed again with the same breakages and the manifest records how many were caught after the feedback. The number of probes is the team's ([generate] mutation_feedback, three by default; zero disables it), and the probe is an aid: a package whose source the harness cannot read is generated without it. Stage 4's mutation run still measures the final file; this is the loop that makes it stronger before it gets there.

Stage 4: Validation execution

This is the harness's own run of the candidate tests: both versions, sealed, once. It is not the suite's regression run, which happens after acceptance wherever the suite runs (section 5.0). All of it happens in a hermetic sandbox, following the OpenSSF Package Analysis design:

• Container built from a pinned base image, one per ecosystem, with the Hermeto-prefetched dependency graph mounted. No outbound network. No mounted secrets. Read-only source tree except for the test output directory. - Declared needs. Every test says what it needs from its testbed, in four tiers, the ladder autopkgtest's restrictions and Google's test sizes already use: none (the default), localhost, network, machine. Python declares with @pytest.mark.needs("network"), Go with // harness:needs network above the test. The team says what its testbed provides ([sandbox] provides, default none, localhost; the sealed container never provides network). A test that needs more is withheld from the sealed run, recorded as skipped with its need, proposed to the team with the declaration, and the behaviour it targets is accounted "for your environment" (principle 15). A test that declared none and then opened a connection is a defect in the test, pytest-socket's rule: the Python sandbox lays a conftest.py beside the tests that raises HarnessUndeclaredNeed on the first connection, Go tests meet the sealed network and are judged by the error, and the verdict says undeclared need. The machine tier runs under Kata where the cluster has bare-metal nodes for it (nested virtualisation is unsupported and gVisor has no CRI-O handler on OpenShift); everywhere else it is an admitted-unavailable tier and goes to the team like network. - Testbed failure is its own status, at two levels, as autopkgtest's exit 16 and Bazel's REMOTE_FAILURE keep it. Run level: the environment did not install, the run timed out, the container exited with no result, or the module did not build for a reason no test is blamed for. Then results.json carries testbed_failure, every test reads not run, the story opens with it, triage raises a testbed-failure finding for the harness's operators, and the accounting puts every reached behaviour in a stated gap with that reason. Test level: one test's failure reads as the environment refusing something (a denied permission, a read-only file system, a missing device, exhausted memory), or a test that declared network or machine and ran met a network error; its verdict is testbed failure, its class testbed-failure, neither a defect nor a test bug. An undeclared connection stays the test's own defect. Export: a CTRF report beside the results, where a testbed failure is other with the harness class in extra, never failed; in JUnit terms it is <error type="testbed-failure">. - CPU, memory, and wall-clock limits per run. Fuzzing gets a fixed time budget.

Validation steps, in order:

1. Compile or import. Tests that do not build are sent back to the generator with the error, up to N retries. Persistent build failures are logged and dropped. 2. Run. Record pass, fail, or error per test with full output. 3. Flake check. Every test runs twice on head, the run and its re-run. The candidates that agreed run again, with only those tests in the sandbox, until they have run as many times as the team set ([sandbox] repeats). Five is the default, TestGen-LLM's published rule: "a test that does not pass on every one of five executions is deemed flaky." Any disagreement quarantines the test; the record carries how many times it ran and how often it disagreed, and the CTRF export carries them as retries and flaky. 4. Coverage. Measure line and branch coverage of the target package attributable to generated tests. 5. Mutation testing. Run the ecosystem's mutation tool (document 05) with a bounded mutant sample drawn only from lines the generated tests executed; a file the tests never reached contributes no site (GF-028). Tests that kill no mutants are marked weak. 6. Differential run. For updates, run the same test set against old and new versions and diff results. 7. Relevance check. For every existing test in scope, generated or human-written, record which symbols and lines it exercises (per-test coverage plus the call graph). Compare that footprint against the API diff and source diff. Tests whose targets no longer exist, whose asserted behavior changed by documented intent, or whose footprint is fully covered by other tests are candidates for retirement. Details in section 8.4.

Stage 5: Triage

The agent classifies every failure and notable finding, in the style of sast-ai-workflow: evidence gathered, verdict with confidence, reflection pass.

Anything under a confidence threshold is escalated rather than auto-routed. Failure logs are summarized with a Log Detective style pass before the classifier sees them.

Two finding classes for the pipeline (principle 14). coverage-gap: the application reaches a package in production and no test of its own references it, or the change alters symbols no existing test reaches. The route is the pipeline's owners, and the suggested change names the call sites. pipeline: one finding per fact stage 2 recorded about the CI, security-class where untrusted code can reach secrets or the network, each with the change that corrects it. A repository with no CI definition at all is itself a finding. The packet carries them in their own section; the harness edits no pipeline.

The accounting (principle 15), as implemented. After the verdicts and the pipeline facts, triage writes triage/<package>/accounting.json and the roll-up triage/accounting-summary.json. The unit is a reached behaviour. For a dependency, it is each call-site symbol outside test files that resolves to a public function, method or type, with re-exports resolved to their definition and module imports set aside. For a first-party change, it is each added or changed public symbol. For a rescan, it is every public symbol within the budget, the rest recorded as out-of-budget. A type is judged by its public methods; a type without any is set aside with that reason. The state comes from the unit's own span: lines a kept test executed (from the run's coverage) and mutants sampled and killed inside it. Gap reasons, checked in order: pipeline-cannot-validate, not-in-public-api, executed-not-checked, strength-not-sampled, flaky, failed-on-head, no-test-generated, out-of-budget. The story, the pull request text ("Accounted for" and "Not covered"), the run index, the dashboard and the evaluation table carry the counts. A unit's lines are its closure: its own span plus the private functions and methods it delegates to within the module (analyze/<package>/internals.json, written by the extractors for Python and Go). The behaviour of decode is credited where it lives, in the helpers. The Python sandbox records a coverage context per test function, so execution is attributed to the test that did it, and strength requires that the test which executed the unit is the one that killed a mutant inside its closure. The Go sandbox has no coverage contexts, so the head run gives each passing test its own go test -run with its own profile (compiled once already, seconds each, capped at forty) and folds them into coverage.json in the same shape. Without contexts, as in older runs, any kept killer counts and the record says so. Mutation sampling is stratified: each reached behaviour with a site gets a mutant, round-robin, before the remainder of the package is sampled, so the strength question is asked where the application calls. Still not done, and named so no one assumes it: the published runs have not been re-run under these rules, because no compute time has been spent on them yet. Until they are, most units on the published runs read strength-not-sampled, which is the truth about the sample they were run with (document 18 has the dry run).

Stage 6: Review

Output per work item is a review packet, posted where the build platform already reports (Konflux integration-service, in the example):

• One-page summary: what changed, what was tested, what was found, recommended action.
• The generated tests as a branch or patch against the test overlay repository (section 8).
• Coverage and mutation score deltas.
• Proposed promotions: overlay tests that met the criteria in section 8.3 and should enter the standard suite.
• Proposed retirements: tests classified obsolete or redundant, each with the symbol or diff evidence and the tests that now cover the same ground.
• Full logs and the replay manifest.

• A Conforma attestation that the run happened, what it concluded, and which tests it produced, promoted, or retired.

• The text and the file list of the test pull request, written here whether or not the pipeline's loop goes on to open one, so every run shows what a person would be asked to accept.

Reviewers approve, edit, or reject tests. Approved tests become permanent regression tests for that package and version range, and from then on they run with the suite, not with the harness.

Every run tells its story. The volume of evidence a run produces is the point, but it is not the reader's entry. Each run keeps a story at its top, rewritten after every stage from the run's own records. It says who did the work and who decided, what was tested, why it earned its budget, where it ran and when. It gives the outcome in the packet's plain terms, the decisions the harness made, and the actions it took and did not take by design. It states each UDLM record as one plain sentence and names every file in the run with the reader it is for. Every file the harness writes carries that label, a reader and a reason, in the run's index. The story states nothing the records do not; it is a rendering, never a source.

The packet is the artifact both lifecycles share. In the developer's loop it stays on the developer's branch and the tests they keep go into their own pull request. In the pipeline's loop the harness opens a test pull request from it, under its own bot identity with signed commits, and the packet is the reviewer's brief. Dependency tests go to the overlay repository with the packet and the records. First-party tests go to the repository under test, against the change's own branch so they join the developer's pull request, with the packet and records in the request's text. The request is opened on GitHub or GitLab by tool or API; on any other forge the branch is pushed and the request is a person's to open. Either way tests are added through a pull request; the harness never merges.

The gate reads two things, and either can say no (decision D8). The pipeline's own result, which now includes the harness's tests running in the team's environment, existing suite included; and the harness's verdict: what was validated, what needs the team's environment, and what was not checked, with the reasons. What counts as inadequate is the team's policy, set in their own file: an unchecked behaviour an advisory says must never happen, a strength score under the floor, too many gaps on new code. Both land on the pull request as statuses a branch policy can require (queue Q7). The harness never merges and never decides alone, but when the policy says its result is a negative, the merge waits and the record says why. A neutral conclusion passes branch protection, so nothing meant to hold a merge may report as neutral.

Stage 7: Feedback

• The review gate's decisions (section 5.0), whether made in code review or on a test pull request, are captured: reviewer edits and rejections become labeled examples for prompt and eval improvement. In lifecycle B, harness feedback reads the test pull request back once a person has decided: which tests landed unchanged, which were edited first, which were dropped, who decided and when. Each accepted candidate's intent record gains a requested record (the reviewer's act) and a realized record (the test as it landed, provided by the overlay repository), and a signed acceptance statement names the merge commit and those records. The harness reads; it changes nothing.
• Tests that later catch a real regression are tagged. This is the ultimate quality signal.
• Per-ecosystem metrics (section 10) drive which adapters and prompts get attention.
• Failure register loop. When a run shows the generator failing in a way the register does not already name, the failure becomes a new register entry before any other fix: issue, reason, cause, detector, correction, self-check. The self-check is written first, fails against the current code, and passes once the correction lands. Document 15.

Prompt and model evaluation, the process. The loop that principle 13 requires runs here, and it is not assumed to have happened: it leaves a table.

1. Prompt sets. Every text said to a model (system prompt per language, task per category, the agent's rules) is a part of a named set. v1 is the code's own texts; any other set overrides parts by key from a file under source control. The active set's name and digest go into every manifest. 2. Ground truth. In order of strength: the reviewer's decision per test from the test pull request (stage 7's labeled examples), the sandbox's verdicts (fix proven, pass on head, flake, mutation kills), triage's acceptance. Cost is recorded beside it: model calls and minutes.
2. The bench. One job that has finished analysis, run again under each candidate set with everything else held: same package, same advisories, same sandbox, same model. A model comparison is the same bench with a different endpoint. One variable per run. 4. The table. One line per set, model, language and package, summed over runs, with a one-phrase reading: accepted by a reviewer, proves fixes, characterizes only, not fit. The document generated from the example runs is the public version of that table. 5. The rule. A set or a model becomes a default only by winning the table on the jobs it is meant for. A set that wins on Python and loses on Go is the Go default for nobody. Nothing in the harness ships a prompt change on the strength of one good run read by eye.

The first application of the rule. The kin-openapi job on the 235B, once under v1 and once under go-api-first, a candidate that tells the model to confirm every symbol against the API data before using it. v1: 8 tests, 432 lines covered, mutation 0.6, six behaviour changes for a reviewer. go-api-first: 7 tests, 169 lines, mutation 0.24, one verified advisory submission blocked on both versions. Neither proved an advisory. v1 stayed the default; the intuitive change lost, and the table says so (document 12).

6. Dependency and transitive strategy

6.1 Why transitives need their own strategy

A mid-sized Go or Java service resolves hundreds to low thousands of packages. Testing all of them at full depth on every change is neither affordable nor useful. The strategy is risk-weighted sampling with full coverage of what we actually reach.

6.2 Depth policy

6.3 Risk score inputs

• Reachability: is the package on a call path from our code? (largest weight). CLDK, govulncheck, dep-scan.
• Change size: lines changed, files changed, and whether the API diff tool reports a contract change.
• Sensitivity: does the package handle parsing, crypto, auth, network, serialization, or process execution? Capslock capabilities for Go.
• Upstream signals: maintainer change, release cadence anomaly, new build scripts, new install hooks. OpenSSF Scorecard as a prior.
• Known vulnerabilities: Trustify Dependency Analytics and OSV matches for the version.
• Pre-flight findings: GuardDog or Malicious Packages hits raise the score to maximum and route straight to suspicious.
• History: has this package caused a triage finding before?

The score sets the generation budget (number of tests, fuzz minutes, mutation sample size) for the item.

The team owns every number (decision D9), and the harness plans to the time it is given (D10). Mutants, repeats, tests per file, model time per change, which gaps block, which dependencies get full effort, what the testbed provides: settings in a file in the team's repository, with defaults the harness ships and explains, and the resolved values written into every run (queue Q8). Cost is described by its shape, never by the minutes one lab measured: model time dominates and scales with the inference hardware. At run start the harness measures that hardware and plans candidates, repairs, probes and mutants to fit the team's time budget with slack, highest-value work first, and reports what it did not reach as a stated gap (queue Q9).

6.4 Characterization snapshots

For every package in the graph, including unreachable transitives, the harness maintains a lightweight behavior snapshot. It holds the exported API surface, a hash of each exported function body, and the outputs of any deterministic zero-argument or fixture-driven calls the agent could construct. On update, the snapshot diff is cheap and tells us where to spend real generation effort.

6.5 Differential testing of upgrades

The single highest-value technique for dependencies is running the same test set against old and new versions. It requires no oracle beyond "the old version's behavior," it is cheap, and it surfaces exactly the class of problem that hurts us: silent behavior changes in code we did not write. Debian autopkgtest and Rust Crater prove the model at distro scale. No open tool does it generically for an application. This is the harness's most original contribution and should be built first.

7. Tooling

The tool per concern, with the fallback and the reason, is the decision table in document 05. Each ecosystem adapter implements the interface in blueprint/adapter-interface.md; adding an ecosystem means implementing that interface, not touching the pipeline.

8. Test lifecycle: where tests live, provenance, promotion, and retirement

8.1 Where generated tests live

We cannot commit tests into upstream repositories we do not own. Generated tests for third-party code live in a test overlay repository, organized by ecosystem, package, and version range:

overlays/
  go/
    github.com/example/lib/
      v1.4.x/
        lib_characterization_test.go
        lib_functional_test.go
        fuzz_parse_test.go
        MANIFEST.yaml        # provenance record, see 8.2
  python/
    requests/
      2.32.x/
        ...
standard/
  go/
    github.com/example/lib/
      lib_contract_test.go   # promoted tests, version-range independent where possible
      MANIFEST.yaml

The overlay is mounted into the sandbox alongside the package source at execution time. When a package version changes, the harness first runs the existing overlay and standard tests against the new version (the differential step) before generating anything new. Overlay tests that prove valuable are candidates for upstream contribution, which is a separate and voluntary step.

First-party tests are committed directly to the product repo on the PR branch, clearly labeled as AI-generated in the commit message and file header, and carry the same provenance record.

8.2 Provenance

Every generated test has a provenance record, kept in the manifest next to the test and embedded in the build attestation. Konflux already produces SLSA provenance as in-toto attestations, so the harness adds a predicate rather than inventing a format.

The record is a UDLM record. UDLM (the Unified Data Lifecycle Model) already keeps one record per lifecycle state, provenance per field, a tamper-evident head on every record chained to its predecessor, and knowledge records for vulnerabilities and packages. The harness does not invent a parallel model: it emits UDLM records and lets the attestation be their projection.

A candidate the harness proposes is an intent record; a reviewer's acceptance is a realized record, by a human act; a retired test is deprecated with its reason. The signed in-toto statement names each evidence record's integrity head as a subject, so a consumer verifies the envelope, matches the subject to the record, and walks the record's history. The join runs one way, statement to record.

The record contains:

The attestation is signed with Trusted Artifact Signer. Conforma policies can then require, for example, that every direct dependency at a release has at least one accepted or standard test attached, or that no suspicious finding is open. The SBOM component entry references the tests that characterize it, so a consumer of the SBOM can find the evidence.

Standards first; UDLM is enablement (decision D13). The record a reader must understand is the in-toto statement, the SLSA provenance and the OpenVEX draft. The same evidence is exported as UDLM records, which link it into the wider lifecycle chain the DCM work is building, so one record spans the change, the build and the estate; that export becomes a switch the team sets (queue Q14), on when the chain is wanted.

8.3 Promotion to the standard suite

Overlay tests start as candidate. Reviewer approval makes them accepted. Promotion to standard is proposed by the harness and confirmed by the owning team when a test meets all of:

• Approved by a reviewer and unchanged, or changed only by reviewers, since approval.
• Kills at least one mutant that no other standard test kills, so it adds strength rather than volume.
• Has survived at least two version bumps of its target without becoming flaky or obsolete, or has caught at least one real regression.
• Asserts behavior we depend on, as shown by our call sites, rather than incidental upstream detail.

Standard tests are the starting point for every future run on that package. They are what the harness runs first in the differential step, and what new candidates are measured against for redundancy. The standard suite is where the harness's output stops being "AI-generated tests" and becomes simply "our tests." Human-written tests can enter the same lifecycle and the same manifest, so provenance and relevance tracking cover the whole suite, not only the generated part.

8.4 Retirement: identifying tests that are no longer relevant

Code changes make tests irrelevant in predictable ways. The relevance engine (stage 4, step 7) detects them using data the harness already collects:

Rules for retirement:

• The harness proposes, with evidence: the diff hunk or removed symbol, the other tests that cover the same ground, the mutant history. A human approves.
• A retired test is moved to retired state, not deleted. Its provenance record keeps the reason. It is re-run once more on the next bump as a safety check, then dropped from execution.
• An obsolete test whose target was replaced rather than removed is first handed back to the generator as a rewrite candidate, since the intent may still be valid against the new API.
• Retirement proposals are advisory. They never block a merge.
• The engine runs on human-written tests in scope too, since stale human tests are as costly as stale generated ones, but proposals for human-written tests are routed to the owning team with a lower default priority.

8.5 SLSA compliance targets

The harness's ability to declare and prove provenance is measured against SLSA. The current specification is SLSA v1.2, which defines a Build track (levels 1 to 3), a Source track (levels 1 to 4), and a Verification Summary Attestation format. Konflux states that it supports SLSA Build Level 3. The harness inherits that for anything it produces as a Konflux PipelineRun output, and adds its own attestations on top.

Build track, applied to harness outputs. Overlay tests, standard tests, review packets, draft VEX statements, and test-evidence attestations are all build outputs of a harness PipelineRun.

Source track, applied to the overlay and standard repositories. These repositories are source for every downstream consumer of the tests.

Test-evidence attestation. SLSA build provenance says how an artifact was built. It does not say what was verified about it. The harness adds in-toto attestations using the vetted test-result/v0.1 predicate (result, configuration descriptors, URL, passed and warned and failed test lists), carrying the section 8.2 record through its configuration descriptors, and the vulns predicate for CVE-targeted findings. One test-result attestation per suite run per work item. These are the predicates Conforma policies evaluate. No harness-specific predicate type is needed.

Verification. Conforma verifies the SLSA provenance and the test-evidence predicate against policy and emits a Verification Summary Attestation per artifact. Policies the harness enables, advisory first and enforced by Phase 3:

• Every direct dependency in a release has at least one accepted or standard test attached.
• Every known vulnerability in the graph has a confirmed VEX statement backed by a test or a reachability record.
• No suspicious or security finding is open.
• Every harness output carries Build L3 provenance and a matching test-evidence predicate.

The harness as a link in the chain. The harness's own components are held to the same standard as its outputs. Its code lives in a repository at Source L3 or better. Its container images are built on Konflux with Build L3 provenance. Prompt files are versioned source with the same review rules as code. Model identity and version are recorded in every attestation, with the model's own disclosure referenced in AI-BOM form where the vendor provides one. Agent configuration is linted with harness-eval on every change. A consumer verifying a test-evidence attestation can therefore verify the harness that produced it, not only the test.

Boundaries. The harness does not confer SLSA levels on the third-party packages it tests. Their build provenance is whatever upstream provides. What the harness attests is what it verified about them, and at what level its own verification process operates.

9. Code generation boundaries

The harness generates code in these categories only:

1. Tests (all types in section 5), including rewrites of obsolete tests.
2. Test support code: fixtures, fakes, mocks, builders, golden files.
3. Harness glue: build scripts, container definitions, tmt plans, adapter shims for a new ecosystem.

It does not generate production code of any kind: no fixes, features, refactors, or dependency version changes. Fixes belong to the fix pipeline (section 2.4), which receives the failing test and the upgrade path from here and sends its change back through this harness. Keeping the boundary narrow keeps review tractable, keeps the evidence independent of what it judges, and keeps the harness out of the product's design decisions.

10. Metrics, quality gates, and benchmarks

The extended goal is coverage of behaviour, not lines (decision D12). For every function the application reaches, every result it can produce, the good path, each class of bad input and each boundary, each combination of conditions that changes the outcome, and each thing an advisory says must never happen, has a test that checks it, proven by mutation testing, or the gap is named (stage 2's behaviour worklist is the denominator; queue Q6 the numerator). Line coverage is used the way Google and DO-178C use it, to find what the behaviour tests missed, never as the score. The pilot's exit criteria are in document 22; the standard for work never seen before and the two-sided regression check on reference jobs are decisions D4 and D14 in document 21.

Cost, as measured on the example runs. The unit that travels between environments is tokens for the model and CPU-minutes for everything else; wall-clock time does not travel, so it is shown only to make that point. Three published runs, one package change each, one lab, two models. Examples, not a forecast: price them in your own tokens and throughput, scaled by your pull requests and dependency bumps per month.

Prompt tokens dominate because the API surface and the call sites are resent on every repair attempt, so prompt caching at the endpoint is the first lever on cost; the environment-aware planner (queue Q9) is the second. Every model call records its usage, so a pilot reports real spend per pull request from the first run rather than estimating it. The harness runs beside the pull request pipeline, never in its critical path: it adds nothing to the build, and the gate policy decides whether to wait for the verdict.

Gates: a PR that bumps a dependency cannot merge while a suspicious or security finding is open. Other findings are advisory.

Benchmarks, run before the pilot and on every prompt or model change:

• BUMP (about 570 reproducible breaking Java dependency updates): does the differential step catch them?
• TDD-Bench-Verified (IBM, 449 instances): can generated tests fail before and pass after a fix?
• Hamster findings: do generated tests include realistic fixtures, mocks, and structured inputs?
• eval-dev-quality (Symflower): generation quality across Java, Go, Kotlin.

11. Security and safety

• Untrusted code execution. Third-party source may be malicious. Sandboxes have no network, no secrets, and are discarded after each run. Kernel isolation (Kata) where the cluster provides it; nested virtualisation is unsupported and gVisor has no CRI-O handler on OpenShift, so on a VM-hosted cluster a test that declares it needs a real kernel is handed to the team's pipeline with the declaration (stage 4), never run on a promise.
• Generated tests are untrusted until read. They run before a reviewer sees them, so they run only in the sandbox above, in the developer's loop as much as in the pipeline's (section 5.0). A plain test runner on a workstation is not a substitute.
• Prompt injection. Package source, docs, comments, and metadata are wrapped as data in every prompt. The agent has no tool that can act outside the sandbox, so a successful injection can at most produce a bad test, which validation then discards. Injection attempts are themselves a suspicious finding.
• Secrets. The generator never sees real credentials. Fixtures use obviously fake values. Signing keys live in the pipeline, never in the sandbox, which is also what SLSA Build L3 requires.
• CVE-targeted and negative tests. Exploit-style inputs execute only in the sandbox. Tests for undisclosed or unfixed vulnerabilities are stored with restricted visibility until disclosure, under the Product Security embargo process. Public reproducers are untrusted data and are rewritten, not copied. Draft VEX statements are proposals to Product Security, never published by the harness.
• Licensing. Generated tests are our work product and licensed under the product's license. The harness records the upstream license of every package it reads. Test overlays for copyleft packages are reviewed by legal before any upstream contribution. Hermeto is GPL-3.0-only and is invoked as a separate tool, never vendored or linked. Every other component recommended for direct reuse is Apache-2.0, MIT, BSD, or ISC. Components with no license file (ChatUniTest, PromptFuzz) or under AGPL (Qodo-Cover, Mutahunter, packj) are excluded.
• Provenance. Every generated file carries a header with run ID, model, and prompt hash, and a full provenance record per section 8.2 embedded in the build's in-toto attestation. Every review packet is signed with Trusted Artifact Signer. SBOM entries reference the tests that characterize each package. The harness records its own model provenance in AI-BOM form. Retired tests keep their record so the history of what was once asserted is never lost.
• Data retention. Source code and prompts are not used for model training. Logs are retained per the standard policy.

12. Governance

• Owner: the AI Test Harness team owns the pipeline, adapters, and prompts.
• Reviewers: the owning team of each product repo reviews packets for depth 0 and 1. Supply Chain Security reviews suspicious findings. Product Security reviews security findings.
• Prompt changes go through the same PR review as code, with the benchmark suite run before merge.
• Model upgrades are treated as a dependency bump: run the benchmarks, compare metrics, roll forward or back.
• Escalation: any reviewer can mark a packet as "harness error," which files a bug against the harness and excludes the packet from acceptance-rate metrics.
• Upstream engagement: findings that affect an open source project are reported upstream through the normal responsible disclosure or issue process, not held privately.

13. Phased rollout

The phases, their goals and their exit criteria are document 07, which supersedes the list this section once carried.

14. Risks and mitigations

15. Open questions

1. Is Konflux integration-service the right trigger for every intake class, or do RPM-sourced changes need a Packit trigger instead?
2. Should the harness be proposed as a Konflux or Konveyor community project, given that Konveyor's agent-mesh work already generates characterization tests for migration?
3. Should depth 2 and deeper characterization snapshots run on every bump or on a schedule?
4. Where do review packets for transitive packages go when no product team obviously owns them?
5. What is the compute budget for the pilot, and who approves it?
6. Is a fully open model stack (Granite via RamaLama or vLLM) a requirement, a preference, or not needed?
7. Which kernel-isolated runtime is supported on the target OpenShift clusters?
8. Can we get access to the TPA "exploit intelligence" reachability feature, or should we plan on CLDK and dep-scan alone?
9. Is QualityFlow a starting codebase or a reference only? Its maintainers should be consulted.
10. Should the provenance predicate be proposed upstream to in-toto or SLSA as a test-evidence attestation type, or kept internal until the format settles?
11. Do product teams want the relevance engine to run over their human-written tests from the start, or only over harness-generated tests until trust is established?
12. Does Product Security want harness-drafted VEX statements as input to its existing CSAF process, and what confirmation workflow and turnaround does that require?
13. Is Source L4 for standard/ acceptable to product teams, given it means two human approvals for every promotion, or should standard/ target L3 with a single approval?
14. Resolved in v0.7: use the vetted in-toto test-result/v0.1 predicate extended through configuration descriptors, plus vulns. Remaining question: should the harness propose a v0.2 of test-result upstream with lifecycle and provenance fields, once the extension has been exercised?

16. Future enhancements (documented now, not scheduled)

These two capabilities are recorded so they shape the architecture even though they are not in the phased plan. Both reuse the harness's intake, sandbox, analysis, and provenance. Both are better understood as a sibling track, supply chain threat detection, that shares the harness rather than as an expansion of the harness's scope.

16.1 Verifying that a change's description matches the change

PR titles, commit messages, changelogs, and release notes describe intent. The harness today reads them as untrusted context. The enhancement is to treat the gap between description and diff as a finding in its own right.

A description-fidelity check would:

1. Have the agent describe what the diff actually does from the diff, the API contract diff, and the call graph alone, with the human-written description withheld from the prompt.
2. Compare that description with the stated one and classify: matches; incomplete (the diff does more than stated, for example a "typo fix" that also changes a build script); mismatch (the diff does something other than stated); undisclosed-sensitive (the diff touches authentication, crypto, input validation, CI configuration, install hooks, or test suites without saying so).
3. Verify claims that can be checked mechanically: "added tests" but no test files changed; "no behavior change" but the differential run disagrees; "fixes CVE-X" but the fix-pinning test still fails; "bumps dependency Y" but the lockfile diff shows other packages moved too.
4. For dependency updates, compare release notes against the actual API diff. The walkthrough in Appendix A already shows this catching an undocumented removal.

The classification goes into the review packet and feeds the risk score. A mismatch or undisclosed-sensitive result on a dependency update raises the item to maximum budget and routes to Supply Chain Security. On a first-party PR it goes back to the author and a second reviewer. The check is deterministic where it can be and model-judged where it must be, with confidence attached, and it never blocks a merge on model judgment alone.

The XZ Utils backdoor of 2024 was carried in commits whose descriptions bore no relation to their effect. That is the class of problem this check exists for, and it is why the check must run with the description withheld: an agent that reads a plausible description first will tend to confirm it.

16.2 Detecting malicious or vulnerable changes across first, second, third, and transitive parties

Interpretation, to confirm the intent: a change anywhere in the supply chain may be malicious (intended to cause harm) or vulnerable (harmful by accident). A malicious change may be direct, where the change itself exfiltrates, backdoors or sabotages, or indirect, where it enables harm elsewhere. An indirect change weakens a check, disables a test, alters a build or CI script, introduces a dependency that is itself malicious, or plants a vulnerability that looks accidental. This applies to first-party code (our own, including insider and compromised-account cases), second-party code (partners, contractors, and forks we co-maintain), and third-party and deeper code (direct and transitive open source dependencies). A deliberately introduced vulnerability is the hard case because it is designed to be indistinguishable from an honest mistake.

What the harness already covers: the suspicious class, the pre-flight gates (GuardDog, Malicious Packages, Capslock), sandbox behavioral observation, negative tests, and CVE-targeted tests. Those catch known-bad packages and observable runtime behavior in third-party code.

What is not covered, and what the enhancement adds:

Output is a malicious-indicator finding with confidence and the corroborating deterministic signals listed. Routing follows suspicious: Supply Chain Security for third party and deeper, Product Security for first and second party. A merge blocks only when a deterministic signal corroborates the model's judgment, never on the model alone.

Relationship to existing work: OpenSSF Package Analysis covers third-party behavioral observation and is already the sandbox pattern. Project Lightwell (IBM and Red Hat, May 2026) targets AI-assisted vulnerability discovery and triage at open source scale but is a commercial program, not something the harness can build on. Nothing in the landscape research screens first-party or second-party changes for malice, and nothing does the indirect analysis. This is the most valuable and the most sensitive thing on this list, which is why it is documented now and built later, on top of a harness whose own chain of trust is already established.

18. Appendix A: Example walkthrough

Event: Renovate opens a PR bumping github.com/example/yamlparse from v1.4.2 to v1.5.0 in service foo.

1. Intake runs Hermeto, which resolves the graph and emits the SBOM. Trustify Dependency Analytics reports yamlparse at depth 1, and the bump also pulls github.com/example/unicode-norm v0.3.0 to v0.3.1 at depth 2. govulncheck says yamlparse is reachable from foo/config; unicode-norm is not. GuardDog and Malicious Packages are clean. Capslock shows yamlparse gained no new capabilities. Trustify Dependency Analytics reports that v1.4.2 is affected by a published advisory for unbounded alias expansion in Parse, fixed in v1.5.0.
2. Analysis runs gorelease. yamlparse.Parse gained an option struct; ParseStrict was removed. Release notes mention the option struct, not the removal. Risk score is high: parser, API change, reachable. unicode-norm gets a snapshot-only budget.
3. Generation reads foo/config to see how Parse and ParseStrict are used, writes 14 functional tests mirroring those call patterns, 22 unit tests for Parse edge cases, 9 negative tests (malformed documents, oversized input, deep nesting), one fuzz target, and two CVE-targeted tests: an exposure test that feeds a recursive-alias document through foo/config, and a fix-pinning test.
4. Execution runs the set against v1.4.2: all pass. Against v1.5.0: 3 functional tests fail to compile (ParseStrict gone), 1 unit test fails (duplicate keys now silently overwrite instead of erroring). Fuzzing finds no crashes in the 10-minute budget. gremlins mutation score 71%. The exposure test shows foo/config was reachable and affected on v1.4.2. The fix-pinning test fails on v1.4.2 and passes on v1.5.0.
5. Triage classifies the compile failures as behavior-change (removal is visible in the API diff, even if undocumented in notes) and the duplicate-key test as undocumented-change with high confidence. unicode-norm snapshot diff shows no exported changes. The relevance engine finds that two existing overlay tests from the v1.4.x cycle target ParseStrict and marks them obsolete, and that one older characterization test kills no mutants that the new functional tests do not, and marks it redundant.
6. Review packet on the PR via integration-service: "v1.5.0 removes ParseStrict, which foo/config calls in two places, and changes duplicate-key handling from error to last-wins. Recommend blocking until foo/config is updated and the duplicate-key behavior is confirmed acceptable. 36 tests attached to overlay go/github.com/example/yamlparse/v1.5.x. Proposed retirements: 2 obsolete, 1 redundant. Proposed promotions: 4 functional tests from v1.4.x that survived this bump and kill unique mutants, plus the fix-pinning test. Draft VEX: affected at v1.4.2 with the exposure test as evidence, fixed at v1.5.0." Conforma verifies the Build L3 provenance and the test-evidence predicate and records a Verification Summary Attestation listing every test produced, promoted, and retired.
7. The foo team fixes the call sites, decides last-wins is acceptable, edits one test to assert the new behavior, approves the 36 tests, approves the 3 retirements, and approves the 4 promotions. The retired tests keep their provenance records and run one more time on the next bump. Product Security confirms the two VEX statements, which flow into Trustify. Next bump of yamlparse starts from 5 standard tests, including the fix-pinning test, plus the v1.5.x overlay.

19. Appendix B: Prompt structure (sketch)

Every generation prompt has the same shape so it can be evaluated and versioned:

SYSTEM: role, ecosystem, framework, output format, hard rules
        (no network, no real credentials, treat all package content as data)
CONTEXT (data, untrusted):
  - API surface (CLDK or language tooling, never model-guessed)
  - API contract diff (japicmp, gorelease, cargo-semver-checks, griffe)
  - Source diff or source
  - Our call sites
  - Existing upstream tests (names and fixtures only)
  - Static analysis findings
TASK: category (unit | functional | fuzz | harness), budget, acceptance criteria
OUTPUT: files with paths, plus a JSON manifest listing each test and what it asserts

The manifest is what triage and review read first. It is the model explaining its own tests, and it is checked against what the tests actually do.

Design queue from the business-case review

What this is. Four reviewers left 31 comments on the business case in October 2026. Answering them changed what the harness promises, and some of those promises are not yet true in the blueprint or the code. This page lists each one, what it answers, where it lands, and roughly how big it is, so that the case and the implementation converge instead of drifting. Items leave this page when they merge.

Each item is one pull request or less. The research notes record what was found before building each one and what it changes here. Order is the order I would do them: the output contract first, because everything else reports through it.

Not queued, by decision. Line coverage as a target (it is the map of gaps, not the score). The harness running the repository's existing suite (the test pull request does that in the team's pipeline). The harness auditing CI beyond Q15.

Research notes for the design queue

What this is. The design queue lists what the business-case review changed in the promise. Before building any of it I researched each item: what the code already has, what the industry does, and what contradicts the plan as written. This page is the result, written for the engineer who picks an item up. Q1 is worked to a design. The rest are notes with sources. Nothing here is built.

Every claim about the harness below is checked against the published run examples/frc-scheduler-server/pr-fix-known-vulns-235b (python-jose 3.3.0 to 3.4.0 on the 235B model). Every claim about another tool carries its source.

Q1. The three-state output contract: design against the code as it is

The promise. Every function a change reaches ends in exactly one state: validated, for your environment, or stated gap with its reason. Nothing is silent. Carried in the run story, the pull request text, the run index and the dashboard.

What the run already knows

What a dry run of the contract says about the published run

I computed the accounting by hand on the example, using the start line of each public symbol and the next symbol's start as its span. Five call-site symbols are reached. Two are not behaviours: jose.jwt is a module import, and jose.JWTError is a re-export whose definition is jose.exceptions.JWTError (the call-site symbol does not match any API symbol, so re-exports need resolving). The three functions:

The four mutants placed in jose/jwt.py sit at lines 260 to 357, inside the private helpers decode calls, and three were killed. So the published run, which the evaluation table reads as "proves fixes: 1 fix-pinning confirmed", would under the contract report zero validated functions and three gaps. That is the honest reading and it exposes two design requirements, not one:

1. Mutant sampling must be stratified over the reached symbols, or almost every reached function will land in "strength not shown" by construction. Today the sample is uniform over the package's 139 sites; 25 draws reached none of the three functions the application calls.
2. A function's strength evidence must include the private helpers it calls. The behaviour of decode lives in _validate_claims and friends. A span bounded by the public symbol misses the lines where breakage would be caught. The unit of strength is the public symbol plus its same-module private callees (a static call closure within the file, which the parser gives cheaply).

The contract, defined

Unit of accounting. A reached behaviour: for a dependency, each distinct call-site symbol outside test files that resolves to a public function, method or class of the package, with re-exports resolved to their definition; a bare module import is not a unit. For a first-party change, each added or changed public symbol. For a rescan with no change, every public symbol, with those beyond the budget recorded as out-of-budget gaps rather than dropped.

States and the evidence each requires.

Reason codes for a gap, in the order they are checked: pipeline-cannot-validate (a P-rule says the repository's CI cannot run or validate the proposed tests), no-test-generated (no kept test names or executes the unit), did-not-build (every candidate for the unit failed to compile or collect), flaky (the only tests for the unit disagreed with themselves), failed-on-head (the tests for the unit are wrong, class test-bug), executed-not-checked (lines ran, no mutant in the closure was killed), strength-not-sampled (lines ran, no mutant was placed in the closure; a sampling failure the harness owns, see requirement 1), out-of-budget. A unit the team's own suite references carries team_suite_references: true as evidence; it does not change the state, because the harness has not measured the team's coverage.

Nothing silent. Every unit appears exactly once. The counts in the roll-up sum to the number of units. A unit with no state is a harness defect and fails the new assessment goal.

Where it lands

• New module stages/account.py, run at the end of triage, since it needs execute, mutate and triage. Writes triage/<pkg>/accounting.json (units with state, reason, evidence) and triage/accounting.json (counts per package and in total).
• model.ApiSymbol gains end_line. Python: ast nodes carry end_lineno since 3.8. Go: the helper has fset.Position(x.End()).Line. Both are one-line changes in the extractors. Private callees within the file come from the same parse.
• Per-test line attribution. Python: coverage.py dynamic contexts, [run] dynamic_context = test_function in a .coveragerc the sandbox writes into /work, then coverage json --show-contexts (the option exists: "Show contexts for covered lines"), which labels each executed line with the tests that ran it. One run, no extra cost. Go: go test -coverprofile is per package; for the kept tests only (a handful), rerun each with -run '^Name$' -count=1 -coverprofile inside the same sandbox. Until this lands, the first iteration attributes execution from the aggregate coverage and strength from killed_by, which is already per test.
• Mutation sampling in mutate.py: allocate the sample across the reached units' closures first, remainder over the package. The team's mutant budget is unchanged; its placement changes.
• Story gains a section "What was accounted for": the three counts, then the gaps with reasons. Pull request text gains a short "Not covered" list (the full report with line notes is Q6). Run index carries the three counts per package; dashboard tiles show validated, for your environment, gaps beside the existing accepted and proven. Catalogue entries for the two new files.
• Assessment gains G13: every reached unit has exactly one state and the counts sum. Blueprint gains principle 15 beside 12 and 13, in the three-page blueprint and the reference; the specification gains R23. The evaluation table gains a "validated" column, which will read lower than "accepted" and should.

Order of work

1. PR A: end_line, resolution of re-exports, account.py from the data that exists, story, pull request text, index, dashboard, G13, principle 15, R23, catalogue, tests on the example runs (every call-site symbol appears exactly once; the python-jose example yields three strength-not-sampled gaps, which is the test that the contract tells the truth).
2. PR B: coverage contexts in the Python sandbox, closures from private callees, stratified mutant sampling. Rerun the python-jose example and publish the before and after.
3. PR C: Go per-test attribution.

What changes in the published numbers

Expect "validated" to be small at first, smaller than "accepted" on every published run, because the sampling never aimed at the reached functions. That is the finding, not a regression: the number the table showed was never the number the case promised. Once sampling is stratified, validated should approach accepted on Python; on Go it stays near zero until the model rung changes, and the contract will say so.

Q2, Q3, Q4. Declared needs, testbed failure, repeat counts

What the industry does. Debian's autopkgtest puts the declaration on the test (needs-internet, isolation-container, isolation-machine, flaky, skippable) and reports a testbed failure at the run level, exit 16, apart from test failures. Bazel does the same split: tags on the target (requires-network, block-network, exclusive, local), statuses PASSED, FLAKY, TIMEOUT, FAILED, REMOTE_FAILURE, exit codes 32 and 36 for environment failures, and a TEST_INFRASTRUCTURE_FAILURE_FILE that "should only be used to report failures originating from the testing infrastructure." tmt puts the isolation choice on the plan, not the test (provision: local | container | virtual | connect), lets tests declare require, recommend, duration, tag and adjust rules, and reports pass, fail, info, warn, error, skip where error means "human inspection is needed to investigate whether it was a test bug, infrastructure error or a real test failure." Testing Farm keeps a request-level error state for provisioning and environment failures, apart from "tests failed."

Enforcement precedents. pytest-socket fails any test that touches a socket when --disable-socket is set, with --allow-hosts for localhost and per-test @pytest.mark.enable_socket. pytest-remotedata marks tests remote_data, skips them by default, and with remote_data_strict fails unmarked tests that reach the internet. Go has no marker: //go:build integration files are invisible to go test without -tags, and t.Skip under testing.Short() is the convention; go test -json carries no skip reason field, the reason is in the preceding output events; always pass -count=1 or repeat runs replay the cache.

Interchange formats. JUnit XML has skipped, failure, error; pytest already routes fixture and setup problems to error and assertion failures to failure. CTRF has passed, failed, skipped, pending, other, plus retries and flaky per test. Neither has an infrastructure-failure status.

Repeat counts. TestGen-LLM: "a test that does not pass on every one of five executions is deemed flaky." Bazel: three attempts for targets marked flaky, one otherwise. CoverUp repeats and prompts the model for a correction when a test is flaky; its default count is undocumented.

Kata and gVisor on OpenShift. The sandboxed containers operator installs RuntimeClass kata; "Nested virtualization is not supported," so Kata needs bare-metal worker nodes; kata-remote offloads to a cloud VM and "is not yet fully supported." gVisor has no CRI-O handler on OpenShift and no Red Hat support, though its systrap platform would run on VM workers. A pod naming a RuntimeClass that does not exist enters Failed with an event before any test code runs. Tekton podTemplate supports runtimeClassName.

What this changes in the plan.

1. Split "the environment refused" into two events. Refusal of a declared need the harness said it would provide is a testbed failure. Use of an undeclared need is a defect in the test's declaration, which is what pytest-socket and pytest-remotedata do, and the test fails. The design sentence "a test that fails because the testbed refused something is never a defect" would otherwise turn every undeclared network call into a non-defect.
2. Testbed failure lives at the run level in every precedent. Keep testbed-failure as the harness's internal class, and define its export: JUnit <error type="testbed-failure">, CTRF other with a label, and a run-level flag in the results and the story. Do not expect consumers to tell it apart on their own.
3. "Skip with a structured reason" is the interoperable answer when the need is known up front. If the sandbox knows it cannot honour network or machine, report skipped with the reason and route to "for your environment." Reserve testbed-failure for "we tried to provide it and it broke."
4. The machine tier has no supported runtime on a VM-hosted OpenShift cluster. Define it either as a VM provisioned outside the cluster (tmt virtual or connect, Testing Farm's Artemis model) or as an admitted-unavailable tier that yields "for your environment" by construction on such clusters. Fail fast at scheduling with the RuntimeClass nodeSelector rather than at sandbox start.
5. Five repeats before proposal, matching the published Meta number; carry retries and flaky in the per-test record so the repeat result exports in CTRF's own fields.
6. For Go, prefer t.Skip guarded by an environment check over build tags for any test the ledger must see, and parse output events for the reason.

Sources: autopkgtest spec, autopkgtest manual, Bazel test encyclopedia, Bazel test status, Bazel exit codes, tmt tests, tmt results, tmt provision, Testing Farm results, pytest-socket, pytest-remotedata, pytest skipping, pytest junitxml source, go build constraints, test2json, CTRF status, JUnit XML, TestGen-LLM, CoverUp, OpenShift sandboxed containers, Kubernetes RuntimeClass, gVisor platforms, Tekton pod templates.

Q6, Q7, Q12. Reporting on the pull request, the gate, reviewer time

GitHub. Check runs carry up to 50 annotations per request (path, start_line, end_line, annotation_level notice/warning/failure, message up to 64 KB), more by updating the run; conclusions success, failure, neutral, skipped, action_required, timed_out, cancelled, stale. "Write permission for the REST API to interact with checks is only available to GitHub Apps"; a token can only post commit statuses (error, failure, pending, success, no annotations). Inside Actions, GITHUB_TOKEN is an App installation token, so a workflow can create check runs with checks: write; ::error file=,line= workflow commands annotate without the API but cap at 10 per level per step and 50 per job. Branch protection requires checks by name: "Required status checks must have a successful, skipped, or neutral status before collaborators can make changes," and rulesets expect the same App to have "recently submitted a check run." Code scanning accepts any SARIF 2.1.0 but needs Code Security on private repositories, fails only on error, critical or high, and leaves persistent alerts to dismiss.

GitLab. The code quality report (description, check_name, fingerprint, severity, location.path, location.lines.begin) shows in the MR widget on all tiers and as diff-gutter markers on Ultimate only. External status checks are Ultimate only and "fail if they stay in the pending state for more than two minutes." On every tier the enforceable gate is "Pipelines must succeed," which "requires the existence of a successful pipeline," so the harness job must itself fail the pipeline.

Tekton and Konflux. Task results are limited to 4 KB by the termination message, 1.5 MB with sidecar logs. Konflux ADR-0030 defines the TEST_OUTPUT result: result in SUCCESS, FAILURE, WARNING, SKIPPED, ERROR plus successes, failures, warnings, note. The integration service reports each scenario's status to the forge as a check run or commit status, pass or fail, and WARNING passes. Conforma reads TEST_OUTPUT from the build attestation or signed test-result attestations (predicate.result, predicate.timestamp, predicate.configuration[0].name) at release time, with rules no_failed_tests, no_erred_tests, no_skipped_tests.

Reviewer time. GitHub: the issue timeline carries reviewed (with submitted_at), review_requested, ready_for_review and convert_to_draft events; the pull request has created_at and merged_at. GitLab: notes with created_at and system, approvals with approved_at, reviewer state, and resource state events. Vendors disagree on the start: LinearB measures pickup from PR creation and review time from first review activity to merge; Swarmia from review request to first review; Faros from ready-for-review to approval. Pick one and state it.

Precedents for a "not covered" view. Codecov posts codecov/patch ("only measures lines adjusted in the pull request") with target, threshold and informational, and a comment of the form "Patch coverage is 76.3% with 14 lines in your changes missing coverage" plus a files table. Coveralls lists "NEW UNCOVERED LINES." SonarQube's gate applies only new-code conditions in pull request analysis and decorates with a status, a summary comment and inline annotations. None of them annotate code outside the diff.

What this changes in the plan.

1. The harness needs a GitHub App identity to annotate, or must ride the Actions token when it runs as a workflow. From Tekton or Konflux it calls the Checks API itself with App credentials or falls back to commit statuses and a comment. This is the bot account question already on Chris's list, now with a reason.
2. "For your environment" cannot map to neutral if it is meant to hold a merge. Neutral and skipped both pass branch protection. The verdict check's conclusion is success or failure as the team's policy decides; the three states go in the summary text.
3. One stable check name per repository, for example ai-test-harness / verdict, or nothing can require it.
4. Annotate within the diff first. Every precedent scopes to changed lines, and GitHub shows annotations outside the diff only on the Checks tab. The "Not covered" section in the pull request body carries the whole list; annotations go on the lines of unchecked behaviours that are inside the change, in batches of 50.
5. Print the threshold and the measured value in the same sentence in the check summary, the way Codecov's status does, to avoid the "red cross but green gate" confusion documented in its issues.
6. On GitLab Free and Premium the harness job fails the pipeline itself; the code quality artifact is the inline channel and works on every tier for the widget.
7. In Konflux, emit TEST_OUTPUT in the ADR-0030 shape so Conforma can gate at release, and post the full verdict to the forge from the task.
8. Reviewer time definition: from the harness's pull request becoming ready for review to the first review activity (pickup), and to merge or close (review time), with bots excluded, which is Swarmia's definition with LinearB's end point. Computed from the timeline in feedback.py.

Sources: GitHub check runs, using the Checks API, commit statuses, protected branches, workflow commands, SARIF support, issue timeline, GitLab code quality, GitLab status checks, GitLab merge checks, Tekton tasks, Konflux ADR-0030, Konflux integration service, Conforma release.test, Conforma test attestations, Codecov statuses, Coveralls notifications, SonarQube quality gates, LinearB review time, Swarmia time to review.

Q5, Q13, Q17. Behaviour worklist, reachability, oracle quality

Nothing off the shelf enumerates behaviours. No tool in Python or Go produces a per-function list of return paths, error paths and parameter ranges. The parsers give the nodes; the enumeration is ours to write, which the plan already assumed. What the parsers give: Python ast has Return, Raise, FunctionDef.args with annotations, Try and ExceptHandler, and end_lineno on every node since 3.8. Go has ast.ReturnStmt, FuncType.Results, and go/types signatures; go/ssa reduces every function to blocks ending in If, Jump, Return or Panic, which is the return and error path inventory at IR level.

Where the parser falls short. Python ast misses implicit None fall-through, exceptions raised by callees, and raises inside with blocks and comprehensions; codeanalyzer-python's control-flow graph at analysis level 3 models raises and is the better base than a hand-rolled visitor. Parameter ranges are not in Python source: only Annotated[int, Ge(0), Le(100)] with annotated-types markers carries bounds, and few codebases use it. So the Python "range" column is the declared type plus the boundaries the type's domain implies (zero, negative, empty, None, maximum), not something read from source. Go types carry no ranges either.

Reachability, Go. govulncheck's recipe is documented and reproducible from golang.org/x/tools: ssa.NewProgram with InstantiateGenerics, cha.CallGraph as the initial graph, vta.CallGraph over it, then a second vta pass, then DeleteSyntheticNodes. Entry points: for main packages only main and init; for library packages, exported functions; generic entry functions are skipped (open TODO). govulncheck itself exposes only findings and call stacks for vulnerable symbols, not the graph, and its graph package is internal, so the recipe must be rebuilt; vta is "in experimental phase." Reflection and function pointers are handled conservatively.

Reachability, Python. Every precise call-graph generator is dead: PyCG archived November 2023 (its paper claimed 99 percent precision and 70 percent recall, so it missed edges even when alive); the Jarvis fork archived March 2026; pytype archived, last supported Python 3.12. CodeLLM-Devkit's Python backend (Jedi plus a def-use pass, "sound-leaning and over-approximate") is the only maintained option beyond call sites and import roots. CLDK is Apache-2.0, release 1.5.0 July 2026, offline, and covers Java, Python and TypeScript; Go support is an open issue from September 2026 with a contributor guide and no backend. OWASP dep-scan lists Go reachability through its golem plugin (go/packages with SSA), Python through atom; atom's own slice spec lists Java, JavaScript and TypeScript for reachables and says "repeated runs could yield slightly varying results." Neither publishes precision or recall.

Checked coverage. Defined as "the dynamic slice of covered statements that actually influence an oracle," computed by backward dynamic slicing from each assertion with JavaSlicer. Java only, including the 2023 host-checked-coverage extension; no Python or Go slicer exists, and building one means a tracer-based dependency tracker. The honest substitute with evidence behind it is state field coverage (Molina, Aguirre, Gorla, ASE 2025): the share of an object's fields an oracle may read, statically computed from the test's AST and the class's field list, which "strongly correlates with the oracles' fault-detection ability, measured by mutation score." Assertion density is cheap but measures production asserts, not test oracles.

Condition and boundary enumeration. Hypothesis from_type builds strategies from annotations and honours annotated-types bounds; its ghostwriter emits starting-point tests. CrossHair cover uses Z3 to find inputs that distinguish paths from type annotations, the closest thing to automatic partitioning for Python. For Go, rapid biases generation to edge cases and gomcdc (released this month) measures unique-cause and masking MC/DC from Go tests. No Python MC/DC tool works from source; py-mcdc takes hand-written boolean expressions.

What this changes in the plan.

1. Behaviour worklist, Python: build on ast for signatures, explicit returns and raises, plus a control-flow pass (Scalpel's or codeanalyzer-python's) for implicit paths; bounds come from the type's domain, labelled as inferred. Go: go/types for signatures, go/ssa for Return and Panic sites and error-result paths.
2. Reachability, Go: reimplement govulncheck's cha-then-vta recipe from x/tools in the Go helper, restricting entry points to main packages for applications. Python: call sites and import roots stay the primary signal; CLDK's over-approximate graph is the one upgrade worth trying, and it should be measured against call sites on the reference jobs before it is trusted.
3. Do not promise checked coverage. Q17 becomes "oracle strength measure": mutation score as the primary, state field coverage as the cheap static companion, both labelled for what they are.
4. Boundary and partition tests: give the model the type-domain boundaries as inputs to try, and consider CrossHair cover as a path-input generator for Python functions with annotations.

Sources: CLDK, CLDK Go issue, codeanalyzer-python, Python ast, PyCG, Jarvis fork, pytype, Scalpel, go/callgraph, vta, govulncheck internals, govulncheck entry points, atom slices, dep-scan reachability, checked coverage, oracle metrics survey, host checked coverage, state field coverage, Hypothesis strategies, CrossHair cover, rapid, gomcdc.

Q8, Q9, Q11. The team's configuration, planning to a budget, reference jobs

Configuration in the team's repository. Every surveyed tool keeps a file in the repository, and none of them puts defaults or reasons in it. The file holds overrides; defaults live in the vendor's documentation and a published schema. Renovate is the strongest model: fourteen accepted file locations, a documentation table per option with type and default, a published JSON schema, a validator (renovate-config-validator, non-zero exit on warnings), presets (extends: ["config:recommended"]) and automated "config migration" pull requests when the format changes. Codecov's codecov.yml separates a project status from a patch status ("only measures lines adjusted in the pull request"), each with target, threshold and informational. Dependabot has no official schema after an issue open since 2020. SonarQube keeps the quality gate server-side and read-only; the repository file carries analysis parameters only. Allstar puts policy at the organisation with repository opt-out, the opposite direction from "the team owns every number."

Measuring throughput. OpenAI-compatible responses carry token counts (usage, and with stream_options.include_usage a final usage chunk) and no timing; the client owns the clock. llama.cpp adds a timings object per response (prompt_per_second, predicted_per_second) and /metrics gauges that average over all slots. vLLM exposes Prometheus histograms (time_to_first_token_seconds, request_time_per_output_token_seconds) and, behind a flag, per-request tokens_per_second, null when n > 1. On OpenShift AI the vLLM runtime is annotated for scraping but only works with user workload monitoring enabled.

Planning to a budget. No CI tool measures model throughput and scales work; the planner is novel. The closest precedent is mutation testing: Stryker and PIT run the suite once and derive per-mutant timeouts as measured time times a factor plus a constant (Stryker 1.5 and 5 seconds, PIT 1.25 and 4 seconds). Everything else is a fixed cap: Bazel's four size buckets, CIFuzz's 600-second default that "should scale with project size," go test's ten minutes, EvoSuite's 60-second search budget. Hypothesis turns its wall-clock deadline off in CI because timing there is noisy.

Reference jobs. For Python and Go with a vulnerable version, a fixed version and a test that fails then passes: Vul4Py (100 Python vulnerabilities, each with an exploit oracle that must fail then pass and a project-native functional oracle that must pass on both; the pair rejected 15 of 119 patches an exploit-only oracle accepted) and PatchEval (1,000 CVEs in Go, JavaScript and Python, 230 with sandboxes for security and functionality tests; the only Go source found). Vul4J and Defects4J are Java; ARVO and SEC-bench are C and C++; CVEfixes has fix commits and no tests; BugsInPy is functional bugs, and only 67 percent reproduced three years on. TDD-Bench Verified (449 Python instances, Apache-2.0) scores fail-to-pass plus coverage of the gold change. Benchmarks decay and leak: Claude Code solved 851 of 854 Defects4J bugs with leakage through cached dependencies and pre-installed reference patches, and 62 percent of its test-adequate patches differed from the developer's fix; SWE-bench-Live exists because SWE-bench is in training data.

What this changes in the plan.

1. Q8: the team's file holds overrides, as everywhere else. The harness ships a schema with default and description per field (Renovate style), a validator, and writes a resolved configuration into every run showing each effective value, its source (default or the team's file) and the reason. That resolved file is where "every number, its default and the reason" lives, and the run story links it. When a default changes, the harness opens a migration pull request the way Renovate does.
2. Q9: the probe times single-sample streamed requests itself and reads the server's own timings or per-request metrics when offered; it must probe at the concurrency it will use, since server averages hide it. The plan applies slack the way Stryker and PIT do, measured times a factor plus a constant, rather than fitting the budget exactly, and the budget is the team's number, as CIFuzz leaves it.
3. Q11: a reference job's known answer is two-sided, fail-to-pass and pass-to-pass, or it will accept regressions. Score against the behavioural oracle, not patch equality. Expect decay: rebuild the reference jobs' environments as sealed images and re-verify them on a schedule. Seed the pool from Vul4Py and PatchEval for Python and Go, and let pilot teams add their own on their own services, which are the only jobs guaranteed not to be in any model's training data.

Sources: Renovate options, Renovate presets, Renovate validator, Codecov statuses, Dependabot options, Dependabot schema issue, SonarQube analysis parameters, Allstar, OpenAI streaming, llama.cpp server, vLLM metrics, vLLM per-request metrics, OpenShift AI model serving, Stryker configuration, PIT options, CIFuzz, Hypothesis settings, Vul4Py, PatchEval, TDD-Bench Verified, Vul4J, CVEfixes, BugsInPy reproduction, Defects4J saturation, SWE-bench-Live.

What the research changes in the queue

Changes to make in the design queue once Chris agrees:

Use case: verified fix ingestion

Status: Proposal, 2026-10-14. An extension of the blueprint at v0.28; it changes nothing the blueprint says the harness is, and it adds one trigger, two verdicts and one relationship. Owner: Chris Roadfeldt. For: the people who decide whether to build it, and the engineers who would.

The problem

When a publisher says a vulnerability is fixed, we take its word. The statement arrives as a security advisory, a VEX document or an advisory database entry: "fixed in version X." Our scanners stop flagging the package once we move to X, and the matter closes. Nobody runs anything. The fix may be incomplete. The fixed version may not resolve into our product the way the publisher built it. The fix may land in a code path our product never reaches while the one we do reach is still open. In each case we find out the way we always have: later, from someone else.

The harness already has the one primitive this needs. A fix is proven by a test that fails on the vulnerable version and passes on the fixed one. A publisher's "fixed" claim is exactly the hypothesis that test is built to check.

The use case in one paragraph

A publisher, an upstream project, a distribution or a vendor, publishes a claim that a vulnerability is fixed in an artifact. The harness reads the claim, checks the artifact's provenance, and finds every product of ours that ships the vulnerable version. For each one it writes and runs the tests that would prove the claim in our environment against our usage. An exposure test runs at our call site, and a fix-pinning test fails on the version we ship and passes on the claimed fix. Each claim ends in one verdict with evidence, and each verdict becomes our own VEX statement, signed, referencing the publisher's. A claim we cannot confirm is disputed with a reproducer, not quietly accepted.

What arrives, and what we do first

Inputs. Claims in the formats publishers already use: CSAF VEX documents, OpenVEX statements, and advisory database entries that carry fixed ranges (OSV and the ecosystem feeds behind it). Each names a package, a vulnerability, a vulnerable range and a fixed version or artifact. Optionally the artifact itself, with provenance: a signed in-toto or SLSA attestation, a Sigstore signature, or a distribution's signed metadata.

Trust before verification. Before anything runs, the harness checks what it can about the claim's origin: the document's signature, the artifact's signature and provenance, the publisher's identity against the list the team trusts. The result is recorded on the claim as verified, unverified or failed, and it travels with every verdict. A claim with failed provenance is still processed, so that we know what it would have proven, but its verdict carries the label and the gate policy can treat it accordingly. The harness never turns an unverified input into a verified-looking output; that is principle 10.

How it maps onto the pipeline

Everything below is the existing pipeline with a different trigger. The stages, the sandbox, the differential run, the records and the rules are the blueprint's.

The verdicts

Every claim ends in exactly one, with the evidence that produced it. Nothing is silent.

The publisher's own statement is never republished as ours. Our statement says what we ran, where, and what happened, and it names theirs.

Two decisions, kept apart. A verdict never admits or refuses the artifact; the harness only tests, and the gate policy says what each verdict is worth. The policy I propose rests on a split. Admitting the artifact is a version change like any other, judged on the ordinary evidence from lifecycle A: the tests on the behaviours our code reaches, run on both versions. Closing the vulnerability is a separate decision, and only a confirmed verdict earns it. So a not-reachable claim does not hold a routine bump back, since the vulnerability was never our exposure, and a could-not-verify claim does not block the artifact either, but our status stays affected or under investigation until a test proves otherwise. An unreachable or unverifiable claim never becomes a "fixed" statement from us.

Where it runs, and how often

As a scheduled sweep, not only on demand. The feeds the team follows are read on a cadence the team sets; each new or changed claim is matched against the products whose SBOMs are in the store; each match is a run. One claim about a widely used package becomes many runs, one per product, because the answer is per product: the same fix can be confirmed for one and disputed for another that pins a different dependent. Effort follows risk as everywhere else: a product that does not reach the vulnerable code gets the reachability verdict without a generation budget.

High-level implementation

What has to be built, in the order I would build it. Sizes are the queue's: small is a day or two, medium a week, large more.

Nothing in generation, execution, mutation, the sandbox, the accounting, the records or the self-checks changes. The register grows only if a run shows a new way to fail.

What does not change

• The harness only tests. It does not apply the publisher's fix, does not move the product to the fixed version, and does not publish a VEX statement; Product Security confirms ours as it confirms every other.
• The boundary with the fix pipeline holds. A disputed claim hands the fix pipeline the same thing a vulnerability does: a failing test with both runs kept, and the publisher's claim as context.
• The fifteen principles hold unchanged. The one this use case leans on hardest is the tenth: the chain of trust is extended, never laundered, so an unverified claim never becomes a verified-looking statement.

Required changes to the blueprint, if adopted

Three lines, none of them a change of design. They are listed here and not applied, because the blueprint stands at v0.28 until this proposal is accepted.

1. Scope. One row in the table of what is in scope and what triggers it: "A published fix claim, from a publisher's advisory, VEX or database feed; on publication or on a scheduled sweep."
2. Triage classes. claim-confirmed and claim-disputed added to the list.
3. Known vulnerabilities become evidence. One sentence: a publisher's fixed claim is verified the same way, by a fix-pinning test in our environment, and our VEX statement names the claim it answers.

The specification would gain one requirement: a claim MUST end in exactly one of the five verdicts with its evidence and provenance label, and our VEX statement MUST reference the publisher's.

Risks and limits, stated

• The model cannot always build the trigger. The register's GF-012 is the honest end of some claims: "could not verify" with the reason, never a default to trust. The rate is a fitness measure for the model and prompt set, on the evaluation table like every other.
• The claimed fix may not resolve into our product. The fixed-candidate path reports what had to move and what would not; that is a disputed claim with a precise reason, and often the more useful finding.
• Reachability is approximate, especially for Python. A "not reachable" verdict carries the evidence and says where it is inferred, and the team's policy decides whether an inferred not-reachable closes a claim.
• Claims without provenance are the common case today. The label is the honest answer; refusing to process them would hide the evidence we can still produce.
• Volume. One claim can fan out to every product that ships the package. The scheduled sweep, the reachability verdict without a generation budget, and the planner (queue Q9) keep it bounded; the dashboard shows the queue.

How it would be judged

• Every claim in the followed feeds has a verdict within the sweep cadence, and no claim is silent.
• The disputed rate per publisher, over time; a dispute the publisher later confirms is the use case paying for itself.
• Time from a claim's publication to our verdict, per product.
• The share of our VEX statements backed by a test that ran, against the share that rest on reachability alone.

A pilot shape

One ecosystem, one SBOM store, one publisher feed the team already reads, and the products that ship the ten most common packages in that feed. Eight weeks after the harness pilot in the case, since it reuses the pilot's model endpoint, sandbox image and bot identity. Exit: every claim in the feed for those packages has a verdict, at least one claim disputed with a reproducer a person confirmed, and Product Security has accepted at least one of our statements as the basis for theirs.

Engineering handoff

Status: 2026-10-14, against the blueprint at v0.28 and the implementation on main. For: the engineering team that takes this on. Two pages: what exists, what it maps to, how to run it, how to extend it, what it needs from you, and what is known not to work.

What you are taking on

A blueprint and a working example implementation of it. The blueprint is the three-page document 03, with the specification listing the twenty-seven things an implementation must do and the test that proves each. The implementation under harness/ does all twenty-seven, for Python and Go, on a laptop with Podman and as a Tekton pipeline on Kubernetes. It has run end to end on two real repositories; every run is under examples/ and on the site, and nothing there is written by hand.

Treat the implementation as the proof of the blueprint, not as the product. It is honest about what it has not done, and the test suite is the contract: 130 tests, green on every merge.

Stage by stage: what exists, where, and what proves it