source: pr-fix-known-vulns-cluster-run2
What happened, why, and what came out of it. Written by the harness from its own records; a person decides what to do with it. The same story opens the run's folder as README.md; the example's write-up is on its example page.
repository sourcelanguage pythonmode diffbase 2ad04243861bhead cfa8f4a3292bmodel qwen/qwen3.8-27bstarted 2026-09-13T23:06:20+00:00harness 0.1.0run 5f493a6b7288
python-jose. This change updates python-jose from 3.3.0 to 3.4.0, which closes 3 known vulnerabilities. The harness could not prove any of the 3 with a test; that is stated, not hidden. Accept the 0 candidate tests if they look right, act on the findings below, and send the draft VEX statements to Product Security.
Who, what, why, where, when
| Who | The AI Test Harness 0.1.0 did the work without a person in the loop. A person decides the result; nothing in this run has been merged. |
| What | Tests for python-jose 3.3.0 to 3.4.0, chosen from a work list of 64 packages, 5 of which this change touched; 7 other package(s) were analyzed and scored but not selected for generation. Repository source, change 2ad04243861b..cfa8f4a3292b, mode diff. |
| Why | python-jose (score 51, budget full): reachable from first-party code; 159 source lines changed; 5 advisories before the change, 0 after. |
| Where | Every test ran in a sealed sandbox (podman, network deny-all networkpolicy on the task pod (claim; verified by the stage 0 probe in this pod), image image-registry.openshift-image-registry.svc:5000/ai-test-harness/harness@sha256:96bf2da89d8bc47f46bc9f8f720c2d98d681ccdeaa2d9a0e9498c0ca2d8b16c1); the model was qwen/qwen3.8-27b in agent mode; the run's own files are in this folder. |
| When | Started 2026-09-13 23:06:20 UTC. The last stage to write was self-verification at 2026-10-13 18:34:44 UTC. |
Decisions the harness made
- python-jose: cut 1 generated test(s) before execution: 1 fails on baseline after repair.
- python-jose: triage decided per test: 1 regenerate or discard; 6 discard as CVE evidence; keep as characterization only if reviewer wants it.
- python-jose: finding, defect at confidence 0.4: python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-33664); NOT corroborated by stage 4 verdicts. Route: escalate: below confidence threshold.
- python-jose: mutation score 0.48 is below the blueprint's 0.6; the packet says so, and the reviewer sees which tests killed nothing.
- python-jose: draft VEX per advisory: CVE-2024-29370 under_investigation, CVE-2024-33663 under_investigation, CVE-2024-33664 under_investigation.
Actions it took
- python-jose: wrote a review packet with 0 accepted test(s) as a patch, ready for a pull request.
- python-jose: drafted VEX statements for Product Security.
- python-jose: signed the test-result statement (PASSED).
- python-jose: no pull request was opened in this run; the packet holds what one would carry.
Actions it did not take, by design
- It did not merge anything. Tests reach a repository only through a pull request that a person merges.
- It did not change production code and did not propose a fix. Fixes come from a separate fix pipeline; this harness only tests.
- It did not publish a VEX statement. The drafts are proposals for Product Security.
- It did not delete any existing test. Retirements are proposed with evidence for a person to act on.
How it happened, stage by stage
What each stage read, did and wrote, with the file behind every number.
Self-verificationstage 0
Read
the harness's own code and the failure register
Did
- 28 register checks, all passed; sandbox probe skipped; model probe skipped.
Intakestage 1
Read
the repository at base and head, its dependency manifest
Did
- Resolved 63 packages at head and 63 at base with ; looked every version up in OSV: 4 with advisories at head; 5 changed; work list of 64 rows.
Analysisstage 2
Read
the work list; package source, advisories, the application's call sites
Did
- 8 package(s) analyzed: ecdsa score 65, budget full, 2 advisories, 0 breaking changes; pdfminer-six score 65, budget full, 4 advisories, 0 breaking changes; pillow score 55, budget full, 0 advisories, 12 breaking changes; pyasn1 score 90, budget full, 8 advisories, 146 breaking changes; python-jose score 51, budget full, 0 advisories, 0 breaking changes; python-multipart score 55, budget full, 0 advisories, 82 breaking changes; starlette score 80, budget full, 14 advisories, 0 breaking changes; weasyprint score 70, budget full, 0 advisories, 152 breaking changes.
Generationstage 3
Read
the fact bundle per package; the model; the sealed sandbox for baseline runs
Did
- Model qwen/qwen3.8-27b, mode agent.
- python-jose: 16 tests kept in 4 file(s), 45 model calls.
Validation executionstage 4
Read
the candidate tests; the environments at head, base or the fixed candidate
Did
- python-jose: sandbox runs new, new-rerun, old on podman (image sha256:96bf2da89d8bc47f46b); 7 tests, 6 pass on head, 0 flaky, 0 fix-pinning confirmed; 543 lines of the package covered.
- Mutation: 118 sites on executed lines, 25 sampled, 12 killed, score 0.48.
- Relevance: 16 generated and 47 application tests examined; 0 obsolete, 0 redundant proposed, horizon this change.
Triagestage 5
Read
every verdict and finding from stage 4
Did
- python-jose: 1 findings, 0 tests to accept, 7 to discard or regenerate, 1 escalated to a person (threshold 0.7).
Packetstage 6
Read
triage, the tests, the advisories
Did
- python-jose: 0 accepted tests in the patch; VEX drafts {"CVE-2024-29370": "under_investigation", "CVE-2024-33663": "under_investigation", "CVE-2024-33664": "under_investigation"}.
Attestationstage attest
Read
the manifest, the tests, the results; UDLM's chain code
Did
- python-jose: 0 provenance records; statement PASSED, key sha256:beb18fb883d5, verified at build; UDLM records sealed, schema problems 0.
Assessmentstage assess
Read
every record the run wrote, against the blueprint's goals
Did
- 10 goals met, 1 not met, 1 not applicable, 1 measured without a target.
Test pull requeststage propose
Read
the packet and the records
Feedbackstage 7
Read
the pull request as a person decided it
python-jose 3.3.0 → 3.4.0
In plain terms. This change updates python-jose from 3.3.0 to 3.4.0, which closes 3 known vulnerabilities. The harness could not prove any of the 3 with a test; that is stated, not hidden. Accept the 0 candidate tests if they look right, act on the findings below, and send the draft VEX statements to Product Security.
Tests, one row each
| Test | Category | Old | New | Verdict | Action |
|---|
tests.test_python_jose_unit | unknown | error | error | fails on head: test bug or defect (new=error) | regenerate or discard |
test_PYSEC_2025_185_fix_pinning | cve | pass | pass | not a fix-pinning test: passes on both versions; keep only as characterization if it covers the symbol | discard as CVE evidence; keep as characterization only if reviewer wants it |
test_PYSEC_2025_185_exposure | cve | pass | pass | not a fix-pinning test: passes on both versions; keep only as characterization if it covers the symbol | discard as CVE evidence; keep as characterization only if reviewer wants it |
test_ghsa_6c5p_j8vq_pqhj_fix_pinning | cve | pass | pass | not a fix-pinning test: passes on both versions; keep only as characterization if it covers the symbol | discard as CVE evidence; keep as characterization only if reviewer wants it |
test_ghsa_6c5p_j8vq_pqhj_exposure | cve | pass | pass | not a fix-pinning test: passes on both versions; keep only as characterization if it covers the symbol | discard as CVE evidence; keep as characterization only if reviewer wants it |
test_ghsa_cjwg_qfpm_7377_fix_pinning | cve | pass | pass | not a fix-pinning test: passes on both versions; keep only as characterization if it covers the symbol | discard as CVE evidence; keep as characterization only if reviewer wants it |
test_ghsa_cjwg_qfpm_7377_exposure | cve | pass | pass | not a fix-pinning test: passes on both versions; keep only as characterization if it covers the symbol | discard as CVE evidence; keep as characterization only if reviewer wants it |
Findings
| Class | Confidence | Finding | Route |
|---|
| defect | 0.4 | python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-33664); NOT corroborated by stage 4 verdicts | escalate: below confidence threshold |
Draft VEX statements, for Product Security
| Vulnerability | Status | Basis |
|---|
| CVE-2024-33663, CVE-2026-85394 | under_investigation | bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet |
| CVE-2024-33664 | under_investigation | bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet |
| CVE-2024-29370 | under_investigation | bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet |
Mutation: which test killed what
118 sites, 25 sampled, 12 killed, score 0.48.
| Test | Mutants killed | Killed only by this test |
|---|
test_PYSEC_2025_185_exposure | 9 | 0 |
test_PYSEC_2025_185_fix_pinning | 9 | 0 |
test_ghsa_6c5p_j8vq_pqhj_exposure | 8 | 3 |
test_ghsa_6c5p_j8vq_pqhj_fix_pinning | 2 | 0 |
test_ghsa_cjwg_qfpm_7377_exposure | 9 | 0 |
test_ghsa_cjwg_qfpm_7377_fix_pinning | 9 | 0 |
The records, in plain terms
9 UDLM records, each a fact from this run in the estate's own language. Intent is what the harness proposed, Requested is what a person was asked to decide, Realized is what landed and who made it so. All are sealed with a content hash.
| Record | State | What it says | Written |
|---|
| Job | Requested | The harness was asked to run on source from 2ad04243861b to cfa8f4a3292b in diff mode. | 2026-09-13 23:06:20 UTC |
| Job | Realized | The run finished; its outputs are the records this statement seals. | 2026-09-14 00:00:33 UTC |
| SoftwarePackage | Discovered | python-jose 3.4.0 (pypi) is the version this change installs. | 2026-09-14 00:00:33 UTC |
| VexStatement | Intent | Draft: CVE-2024-33663 is under_investigation in this package. bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet | 2026-09-14 00:00:33 UTC |
| VexStatement | Intent | Draft: CVE-2024-33664 is under_investigation in this package. bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet | 2026-09-14 00:00:33 UTC |
| VexStatement | Intent | Draft: CVE-2024-29370 is under_investigation in this package. bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet | 2026-09-14 00:00:33 UTC |
| Vulnerability | Discovered | CVE-2024-33663 is a known vulnerability affecting <3.4.0. Also known as GHSA-6c5p-j8vq-pqhj, PYSEC-2024-232. | 2026-09-14 00:00:33 UTC |
| Vulnerability | Discovered | CVE-2024-33664 is a known vulnerability affecting <3.4.0. Also known as GHSA-cjwg-qfpm-7377, PYSEC-2024-233. | 2026-09-14 00:00:33 UTC |
| Vulnerability | Discovered | CVE-2024-29370 is a known vulnerability affecting versions the advisory does not pin. Also known as PYSEC-2025-185. | 2026-09-14 00:00:33 UTC |
The run against the blueprint's goals
| Id | Goal | Measured | Verdict | From |
|---|
| G1 | Nothing merges without a person | harness produced patches and drafts only; no git push, no VEX published | met | packet.md |
| G2 | Every kept test was compiled and run in the sandbox before a reviewer sees it | 7 tests ran on head, again for flakes, and on base; 0 flaky | met | results.json |
| G3 | A test that passes on both versions is not presented as CVE evidence | 0 fix-pinning confirmed; pass-on-both CVE tests are classified test-bug and excluded from VEX 'fixed' | met | triage.json |
| G4 | Stage 0 passed before evidence was produced | 28 of 28 register checks; sandbox probe skipped; model probe skipped | met | selfcheck.json |
| G5 | Generated tests ran with no network, no secrets, read-only root, disposable container | {'disposable': 'the task pod', 'limits': {'cpus': '2', 'memory': '2g', 'pids': 512, 'timeout_s': 900}, 'network': ' | met | sandbox.json |
| G6 | Every accepted test carries a provenance record and a signed attestation | 0 records; envelopes verified locally: [True]; signer is a development key | met (development signer) | statement.dsse.json |
| G7 | Every advisory on the work item has a CVE-targeted test attempt and a draft VEX statement | python-jose: {'CVE-2024-29370': 'under_investigation', 'CVE-2024-33663': 'under_investigation', 'CVE-2024-33664': 'under_investigation'} | met | vex.openvex.json |
| G8 | The run surfaces a finding the PR diff does not show | python-jose: ['python-jose: a code path raised the same internal error on both versions during '] | met | triage.json |
| G9 | Packet within 2 hours of trigger for depth 0 and 1 | intake 2026-09-13T23:06:40+00:00 -> packet 2026-09-14T00:08:59+00:00 | see measurement | run.json summary.json |
| G10 | Every number in the packet points at a file in the work directory | packet sections cite artifact paths; attestation subjects are the patch and the manifest digests | met | statement.json |
| G11 | Test strength measured by mutation and coverage delta | python-jose: score 0.48 on 25 sampled mutants; coverage delta: no baseline overlay suite yet | not met (target 0.6 on sampled mutants) | mutation.json |
| G12 | The pipeline's own gaps are stated, never inherited silently | analysis predates the pipeline facts; rerun analysis | not applicable | |
| G13 | Every behaviour the change reaches ends in exactly one state, and a gap carries its reason | 3 behaviour(s) reached: 0 validated, 0 for your environment, 3 stated gap(s) (3 strength-not-sampled); 2 reference(s) set aside with a reason | met | accounting-summary.json accounting.json |
What is in this folder, and who it is for
What is in this folder, and who it is for
Every file exists for a reader or a tool named here. The plain-English files come first; the machine files stay because they are the proof and the replay.
Everyone
README.md: The story of this run: who, what, why, where, when, the outcome, the decisions and the actions.triage/accounting-summary.json: The accounting rolled up: how many behaviours were reached, validated, handed to the team, or left as stated gaps, and why.assess/assess.md: The run against the blueprint's thirteen goals, each with the measurement and the file it came from.
Reviewers (the developer or team that owns the change)
intake/worklist.json: The work list: every package at base and head, what changed, what is reachable, what carries advisories.analyze/ecdsa/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/pdfminer-six/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/pillow/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/pyasn1/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/python-jose/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/python-multipart/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/starlette/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/weasyprint/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/pillow/api-diff.json: Every added, removed or changed public symbol between the two versions, with a breaking flag.analyze/pyasn1/api-diff.json: Every added, removed or changed public symbol between the two versions, with a breaking flag.analyze/python-jose/api-diff.json: Every added, removed or changed public symbol between the two versions, with a breaking flag.analyze/python-multipart/api-diff.json: Every added, removed or changed public symbol between the two versions, with a breaking flag.analyze/weasyprint/api-diff.json: Every added, removed or changed public symbol between the two versions, with a breaking flag.generate/python-jose/tests/: The candidate test files as generated, before any decision.execute/python-jose/results.json: The verdict per test: outcome on head, on the re-run and on the other version, coverage, and how each was judged.execute/python-jose/mutation/mutation.json: How strong the accepted tests are: which mutants of the package's source they killed, and the score.execute/python-jose/relevance.json: Which existing tests this change makes obsolete or redundant, proposed with evidence; nothing is deleted.triage/python-jose/triage.json: Every verdict and finding classified, with a confidence and a route; below the threshold a person decides.triage/python-jose/accounting.json: The accounting: every behaviour the change reaches in exactly one state, validated, for your environment or a stated gap with its reason, with the evidence per behaviour.packet/python-jose/packet.md: The review packet: the verdict in plain terms, one row per test, the findings, the VEX drafts, the retirements, the recommended action.packet/python-jose/tests.patch: The accepted tests as a patch in the layout they will live in.packet/python-jose/pull-request.md: The text and the file list the test pull request carries, whether or not one was opened.
Product Security
intake/vulns.json: Known vulnerabilities for every package version in either graph, from OSV, including malicious-package reports.analyze/ecdsa/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/pdfminer-six/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/pillow/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/pyasn1/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/python-jose/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/python-multipart/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/starlette/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.analyze/weasyprint/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.packet/python-jose/vex.openvex.json: One draft VEX statement per advisory, with its basis; a proposal for Product Security, never published by the harness.
Supply Chain Security and auditors
run.json: What was run, with which tool versions, on which repository by name.selfcheck/selfcheck.json: Stage 0: every failure-register check and the sandbox and model probes, before any evidence was produced.intake/sbom.new.cdx.json: The software bill of materials at head, CycloneDX.generate/python-jose/model-calls/: Every prompt and every response, numbered, with the call's settings; the full transcript of what the model saw and said.attest/python-jose/MANIFEST.json: The provenance record per accepted test: model, prompt and response digests, sandbox image digest, verdicts.attest/python-jose/statement.json: The in-toto test-result statement; its subjects are the patch, the record and each evidence record's head.attest/python-jose/statement.dsse.json: The statement signed as a DSSE envelope.attest/python-jose/signer.pub.pem: The public key that verifies the envelope.attest/python-jose/attest.json: Which key signed, whether it verified locally, and what the run could not back.attest/python-jose/udlm/: The same facts as sealed UDLM records: the vulnerability, the package, the run, each accepted test, the draft VEX.
The person who ran the harness
analyze/pillow/source-diff.patch: The package's own source diff between the versions; the model reads it to find the fix.analyze/pyasn1/source-diff.patch: The package's own source diff between the versions; the model reads it to find the fix.analyze/python-jose/source-diff.patch: The package's own source diff between the versions; the model reads it to find the fix.analyze/python-multipart/source-diff.patch: The package's own source diff between the versions; the model reads it to find the fix.analyze/weasyprint/source-diff.patch: The package's own source diff between the versions; the model reads it to find the fix.analyze/ecdsa/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/pdfminer-six/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/pillow/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/pyasn1/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/python-jose/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/python-multipart/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/starlette/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.analyze/weasyprint/notes.untrusted.md: Package metadata and description as published upstream; untrusted text, kept for context.generate/python-jose/manifest.json: What was generated: every file, every test, what was cut and discarded and why, the model, the prompt set and its digest, the prompt and response digests.generate/python-jose/manifest.agent.json: The tool-using agent's traces: every tool call per advisory, and the budget it had.
Tools (kept for replay and proof; not meant to be read)
run-index.json: What this run holds: stages, packages, headline numbers, and where every file is; the renderers read it.analyze/ecdsa/api.new.json: The public API surface at head, extracted by tools.analyze/pdfminer-six/api.new.json: The public API surface at head, extracted by tools.analyze/pillow/api.new.json: The public API surface at head, extracted by tools.analyze/pyasn1/api.new.json: The public API surface at head, extracted by tools.analyze/python-jose/api.new.json: The public API surface at head, extracted by tools.analyze/python-multipart/api.new.json: The public API surface at head, extracted by tools.analyze/starlette/api.new.json: The public API surface at head, extracted by tools.analyze/weasyprint/api.new.json: The public API surface at head, extracted by tools.analyze/pillow/api.old.json: The public API surface at base.analyze/pyasn1/api.old.json: The public API surface at base.analyze/python-jose/api.old.json: The public API surface at base.analyze/python-multipart/api.old.json: The public API surface at base.analyze/weasyprint/api.old.json: The public API surface at base.analyze/ecdsa/call-sites.json: Every place the application references this package.analyze/pdfminer-six/call-sites.json: Every place the application references this package.analyze/pillow/call-sites.json: Every place the application references this package.analyze/pyasn1/call-sites.json: Every place the application references this package.analyze/python-jose/call-sites.json: Every place the application references this package.analyze/python-multipart/call-sites.json: Every place the application references this package.analyze/starlette/call-sites.json: Every place the application references this package.analyze/weasyprint/call-sites.json: Every place the application references this package.execute/python-jose/new/: The sealed run on head: the run script, the junit report, coverage, logs.execute/python-jose/new-rerun/: The second run on head, to catch flakes.execute/python-jose/old/: The run on the base version, for the differential.execute/python-jose/mutation/mutants/: Each sampled mutant: its change as a patch, and its sealed run's output.packet/python-jose/packet.json: The packet's facts in structured form.assess/assess.json: The assessment in structured form.