frc-scheduler-server: pr-fix-known-vulns-235b
What happened, why, and what came out of it. Written by the harness from its own records; a person decides what to do with it. The same story opens the run's folder as README.md; the example's write-up is on its example page.
repository frc-scheduler-serverlanguage pythonmode diffbase 2ad04243861bhead cfa8f4a3292bmodel qwen3-235bstarted 2026-09-22T20:11:29+00:00harness 0.1.0run 6a9a03828b7f
python-jose. This change updates python-jose from 3.3.0 to 3.4.0, which closes 3 known vulnerabilities. The harness proved 1 of the 3 with a test that fails on the vulnerable version and passes on the fixed one; the other 2 are unproven and marked so. Accept the 8 candidate tests if they look right, act on the findings below, and send the draft VEX statements to Product Security.
Who, what, why, where, when
| Who | The AI Test Harness 0.1.0 did the work without a person in the loop. A person decides the result; nothing in this run has been merged. |
| What | Tests for python-jose 3.3.0 to 3.4.0, chosen from a work list of 64 packages, 5 of which this change touched. Repository frc-scheduler-server, change 2ad04243861b..cfa8f4a3292b, mode diff. |
| Why | python-jose (score 51, budget full): reachable from first-party code; 159 source lines changed; 5 advisories before the change, 0 after. |
| Where | Every test ran in a sealed sandbox (podman, network none, image docker.io/library/python:3.12-slim); the model was qwen3-235b in agent mode; the run's own files are in this folder. |
| When | Started 2026-09-22 20:11:29 UTC. The last stage to write was self-verification at 2026-10-13 18:34:44 UTC. |
Decisions the harness made
- python-jose: cut 3 generated test(s) before execution: 3 fails on baseline after repair.
- python-jose: discarded 1 generation attempt(s): 1 agent exhausted its budget without an accepted submission.
- python-jose: triage decided per test: 2 open issue; reproducer is the test; advisory stays under investigation; 1 regenerate or discard; 1 accept as CVE evidence; VEX status fixed; 7 accept as characterization candidate.
- python-jose: finding, defect at confidence 0.4: python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-33663); NOT corroborated by stage 4 verdicts. Route: escalate: below confidence threshold.
- python-jose: finding, defect at confidence 0.4: python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-33664); NOT corroborated by stage 4 verdicts. Route: escalate: below confidence threshold.
- python-jose: finding, defect at confidence 0.8: python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-29370). Route: open issue with reproducer.
- python-jose: finding, redundant at confidence 0.6: python-jose: test_jwt_encode_returns_string_with_hs256's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9. Route: escalate: below confidence threshold.
- python-jose: finding, redundant at confidence 0.6: python-jose: test_jwt_decode_with_valid_token_and_key_returns_claims's 8 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9. Route: escalate: below confidence threshold.
- python-jose: finding, redundant at confidence 0.6: python-jose: test_jwt_decode_with_invalid_signature_raises_JWTError's 2 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9. Route: escalate: below confidence threshold.
- python-jose: finding, redundant at confidence 0.6: python-jose: test_jwt_get_unverified_header_returns_headers_dict's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9. Route: escalate: below confidence threshold.
- python-jose: finding, redundant at confidence 0.6: python-jose: test_jwt_get_unverified_claims_returns_payload_dict's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9. Route: escalate: below confidence threshold.
- python-jose: mutation score 0.4 is below the blueprint's 0.6; the packet says so, and the reviewer sees which tests killed nothing.
- python-jose: draft VEX per advisory: CVE-2024-29370 under_investigation, CVE-2024-33663 fixed, CVE-2024-33664 under_investigation.
Actions it took
- python-jose: wrote a review packet with 8 accepted test(s) as a patch, ready for a pull request.
- python-jose: drafted VEX statements for Product Security.
- python-jose: signed the test-result statement (PASSED).
- python-jose: no pull request was opened in this run; the packet holds what one would carry.
Actions it did not take, by design
- It did not merge anything. Tests reach a repository only through a pull request that a person merges.
- It did not change production code and did not propose a fix. Fixes come from a separate fix pipeline; this harness only tests.
- It did not publish a VEX statement. The drafts are proposals for Product Security.
- It did not delete any existing test. Retirements are proposed with evidence for a person to act on.
How it happened, stage by stage
What each stage read, did and wrote, with the file behind every number.
Self-verificationstage 0
Read
the harness's own code and the failure register
Did
- 28 register checks, all passed; sandbox probe skipped; model probe skipped.
Intakestage 1
Read
the repository at base and head, its dependency manifest
Did
- Resolved 63 packages at head and 63 at base with ; looked every version up in OSV: 4 with advisories at head; 5 changed; work list of 64 rows.
Analysisstage 2
Read
the work list; package source, advisories, the application's call sites
Did
- 1 package(s) analyzed: python-jose score 51, budget full, 0 advisories, 0 breaking changes.
Generationstage 3
Read
the fact bundle per package; the model; the sealed sandbox for baseline runs
Did
- Model qwen3-235b, mode agent.
- python-jose: 11 tests kept in 3 file(s), 48 model calls; discarded: cve CVE-2024-33664: agent exhausted its budget without an accepted submission.
Validation executionstage 4
Read
the candidate tests; the environments at head, base or the fixed candidate
Did
- python-jose: sandbox runs new, new-rerun, old on podman (image sha256:2fe5997d249a808b8ee); 11 tests, 8 pass on head, 0 flaky, 1 fix-pinning confirmed; 571 lines of the package covered.
- Mutation: 139 sites on executed lines, 25 sampled, 10 killed, score 0.4.
- Relevance: 11 generated and 47 application tests examined; 0 obsolete, 5 redundant proposed, horizon this change.
Triagestage 5
Read
every verdict and finding from stage 4
Did
- python-jose: 8 findings, 8 tests to accept, 3 to discard or regenerate, 7 escalated to a person (threshold 0.7).
Packetstage 6
Read
triage, the tests, the advisories
Did
- python-jose: 8 accepted tests in the patch; VEX drafts {"CVE-2024-29370": "under_investigation", "CVE-2024-33663": "fixed", "CVE-2024-33664": "under_investigation"}.
Attestationstage attest
Read
the manifest, the tests, the results; UDLM's chain code
Did
- python-jose: 8 provenance records; statement PASSED, key sha256:4f2defc2e375, verified at build; UDLM records sealed, schema problems 0.
Assessmentstage assess
Read
every record the run wrote, against the blueprint's goals
Did
- 10 goals met, 1 not met, 1 not applicable, 1 measured without a target.
Test pull requeststage propose
Read
the packet and the records
Feedbackstage 7
Read
the pull request as a person decided it
python-jose 3.3.0 → 3.4.0
In plain terms. This change updates python-jose from 3.3.0 to 3.4.0, which closes 3 known vulnerabilities. The harness proved 1 of the 3 with a test that fails on the vulnerable version and passes on the fixed one; the other 2 are unproven and marked so. Accept the 8 candidate tests if they look right, act on the findings below, and send the draft VEX statements to Product Security.
Tests, one row each
| Test | Category | Old | New | Verdict | Action |
|---|
test_pysec_2025_185_fix_pinning | cve | fail | fail | blocked on both versions: jose.exceptions.JWEError on old and new; possible package or environment defect, reproducer attached | open issue; reproducer is the test; advisory stays under investigation |
test_pysec_2025_185_exposure | cve | fail | fail | blocked on both versions: jose.exceptions.JWEError on old and new; possible package or environment defect, reproducer attached | open issue; reproducer is the test; advisory stays under investigation |
test_GHSA_6c5p_j8vq_pqhj_fix_pinning | cve | fail | fail | fails on the fixed version: test bug or the advisory is misread; back to generation | regenerate or discard |
test_GHSA_6c5p_j8vq_pqhj_exposure | cve | fail | pass | fix-pinning confirmed: fails on vulnerable, passes on fixed | accept as CVE evidence; VEX status fixed |
test_jwt_encode_returns_string_with_hs256 | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
test_jwt_decode_with_valid_token_and_key_returns_claims | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
test_jwt_decode_with_invalid_signature_raises_JWTError | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
test_jwt_decode_with_expired_token_raises_ExpiredSignatureError | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
test_jwt_get_unverified_header_returns_headers_dict | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
test_jwt_get_unverified_claims_returns_payload_dict | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
test_jwt_decode_with_audience_checks_aud_claim | unit | pass | pass | candidate: passes on head, same on old | accept as characterization candidate |
Findings
| Class | Confidence | Finding | Route |
|---|
| defect | 0.4 | python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-33663); NOT corroborated by stage 4 verdicts | escalate: below confidence threshold |
| defect | 0.4 | python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-33664); NOT corroborated by stage 4 verdicts | escalate: below confidence threshold |
| defect | 0.8 | python-jose: a code path raised the same internal error on both versions during CVE test generation (CVE-2024-29370) | open issue with reproducer |
| redundant | 0.6 | python-jose: test_jwt_encode_returns_string_with_hs256's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 | escalate: below confidence threshold |
| redundant | 0.6 | python-jose: test_jwt_decode_with_valid_token_and_key_returns_claims's 8 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 | escalate: below confidence threshold |
| redundant | 0.6 | python-jose: test_jwt_decode_with_invalid_signature_raises_JWTError's 2 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 | escalate: below confidence threshold |
| redundant | 0.6 | python-jose: test_jwt_get_unverified_header_returns_headers_dict's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 | escalate: below confidence threshold |
| redundant | 0.6 | python-jose: test_jwt_get_unverified_claims_returns_payload_dict's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 | escalate: below confidence threshold |
Draft VEX statements, for Product Security
| Vulnerability | Status | Basis |
|---|
| CVE-2024-33663, CVE-2026-85394 | fixed | fix-pinning test(s) ['test_GHSA_6c5p_j8vq_pqhj_exposure'] fail on pkg:pypi/python-jose@3.3.0 and pass on pkg:pypi/python-jose@3.4.0 |
| CVE-2024-33664 | under_investigation | bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet |
| CVE-2024-29370 | under_investigation | bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet |
Mutation: which test killed what
139 sites, 25 sampled, 10 killed, score 0.4.
| Test | Mutants killed | Killed only by this test |
|---|
test_GHSA_6c5p_j8vq_pqhj_exposure | 2 | 0 |
test_jwt_decode_with_audience_checks_aud_claim | 9 | 1 |
test_jwt_decode_with_expired_token_raises_ExpiredSignatureError | 9 | 1 |
test_jwt_decode_with_invalid_signature_raises_JWTError | 2 | 0 |
test_jwt_decode_with_valid_token_and_key_returns_claims | 8 | 0 |
test_jwt_encode_returns_string_with_hs256 | 5 | 0 |
test_jwt_get_unverified_claims_returns_payload_dict | 5 | 0 |
test_jwt_get_unverified_header_returns_headers_dict | 5 | 0 |
Retirement proposals
| Test | Reason | Evidence |
|---|
test_jwt_encode_returns_string_with_hs256 | redundant-subsumed | test_jwt_encode_returns_string_with_hs256's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 |
test_jwt_decode_with_valid_token_and_key_returns_claims | redundant-subsumed | test_jwt_decode_with_valid_token_and_key_returns_claims's 8 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 |
test_jwt_decode_with_invalid_signature_raises_JWTError | redundant-subsumed | test_jwt_decode_with_invalid_signature_raises_JWTError's 2 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 |
test_jwt_get_unverified_header_returns_headers_dict | redundant-subsumed | test_jwt_get_unverified_header_returns_headers_dict's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 |
test_jwt_get_unverified_claims_returns_payload_dict | redundant-subsumed | test_jwt_get_unverified_claims_returns_payload_dict's 5 mutant kill(s) are all among test_jwt_decode_with_audience_checks_aud_claim's 9 |
Discarded during generation
- cve CVE-2024-33664: agent exhausted its budget without an accepted submission
The records, in plain terms
17 UDLM records, each a fact from this run in the estate's own language. Intent is what the harness proposed, Requested is what a person was asked to decide, Realized is what landed and who made it so. All are sealed with a content hash.
| Record | State | What it says | Written |
|---|
| Job | Requested | The harness was asked to run on frc-scheduler-server from 2ad04243861b to cfa8f4a3292b in diff mode. | 2026-09-22 20:11:29 UTC |
| Job | Realized | The run finished; its outputs are the records this statement seals. | 2026-09-22 22:48:32 UTC |
| SoftwarePackage | Discovered | python-jose 3.4.0 (pypi) is the version this change installs. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_GHSA_6c5p_j8vq_pqhj_exposure is an exposure test for in tests/test_python_jose_cve_cve_2024_33663.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_decode_with_audience_checks_aud_claim is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_decode_with_expired_token_raises_ExpiredSignatureError is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_decode_with_invalid_signature_raises_JWTError is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_decode_with_valid_token_and_key_returns_claims is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_encode_returns_string_with_hs256 is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_get_unverified_claims_returns_payload_dict is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| TestEvidence | Intent | test_jwt_get_unverified_header_returns_headers_dict is a unit test in tests/test_python_jose_unit.py; it passed on head in the sealed sandbox. | 2026-09-22 22:48:32 UTC |
| VexStatement | Intent | Draft: CVE-2024-33663 is fixed in this package. fix-pinning test(s) ['test_GHSA_6c5p_j8vq_pqhj_exposure'] fail on pkg:pypi/python-jose@3.3.0 and pass on pkg:pypi/python-jose@3.4.0 | 2026-09-22 22:48:32 UTC |
| VexStatement | Intent | Draft: CVE-2024-33664 is under_investigation in this package. bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet | 2026-09-22 22:48:32 UTC |
| VexStatement | Intent | Draft: CVE-2024-29370 is under_investigation in this package. bump to a fixed version per advisory metadata; no confirmed fix-pinning test yet | 2026-09-22 22:48:32 UTC |
| Vulnerability | Discovered | CVE-2024-33663 is a known vulnerability affecting <3.4.0. Also known as GHSA-6c5p-j8vq-pqhj, PYSEC-2024-232. | 2026-09-22 22:48:32 UTC |
| Vulnerability | Discovered | CVE-2024-33664 is a known vulnerability affecting <3.4.0. Also known as GHSA-cjwg-qfpm-7377, PYSEC-2024-233. | 2026-09-22 22:48:32 UTC |
| Vulnerability | Discovered | CVE-2024-29370 is a known vulnerability affecting versions the advisory does not pin. Also known as PYSEC-2025-185. | 2026-09-22 22:48:32 UTC |
The run against the blueprint's goals
| Id | Goal | Measured | Verdict | From |
|---|
| G1 | Nothing merges without a person | harness produced patches and drafts only; no git push, no VEX published | met | packet.md |
| G2 | Every kept test was compiled and run in the sandbox before a reviewer sees it | 11 tests ran on head, again for flakes, and on base; 0 flaky | met | results.json |
| G3 | A test that passes on both versions is not presented as CVE evidence | 1 fix-pinning confirmed; pass-on-both CVE tests are classified test-bug and excluded from VEX 'fixed' | met | triage.json |
| G4 | Stage 0 passed before evidence was produced | 28 of 28 register checks; sandbox probe skipped; model probe skipped | met | selfcheck.json |
| G5 | Generated tests ran with no network, no secrets, read-only root, disposable container | {'capabilities': 'all dropped', 'disposable': True, 'limits': {'cpus': '2', 'memory': '2g', 'pids': 512, 'timeout_s': 90 | met | sandbox.json |
| G6 | Every accepted test carries a provenance record and a signed attestation | 8 records; envelopes verified locally: [True]; signer is a development key | met (development signer) | statement.dsse.json |
| G7 | Every advisory on the work item has a CVE-targeted test attempt and a draft VEX statement | python-jose: {'CVE-2024-29370': 'under_investigation', 'CVE-2024-33663': 'fixed', 'CVE-2024-33664': 'under_investigation'} | met | vex.openvex.json |
| G8 | The run surfaces a finding the PR diff does not show | python-jose: ['python-jose: a code path raised the same internal error on both versions during ', 'python-jose: a code path raised the same internal error on both versions during ', 'python-jose: | met | triage.json |
| G9 | Packet within 2 hours of trigger for depth 0 and 1 | intake 2026-09-22T20:12:38+00:00 -> packet 2026-09-22T23:09:03+00:00 | see measurement | run.json summary.json |
| G10 | Every number in the packet points at a file in the work directory | packet sections cite artifact paths; attestation subjects are the patch and the manifest digests | met | statement.json |
| G11 | Test strength measured by mutation and coverage delta | python-jose: score 0.4 on 25 sampled mutants; coverage delta: no baseline overlay suite yet | not met (target 0.6 on sampled mutants) | mutation.json |
| G12 | The pipeline's own gaps are stated, never inherited silently | analysis predates the pipeline facts; rerun analysis | not applicable | |
| G13 | Every behaviour the change reaches ends in exactly one state, and a gap carries its reason | 3 behaviour(s) reached: 0 validated, 0 for your environment, 3 stated gap(s) (3 strength-not-sampled); 2 reference(s) set aside with a reason | met | accounting-summary.json accounting.json |
What is in this folder, and who it is for
What is in this folder, and who it is for
Every file exists for a reader or a tool named here. The plain-English files come first; the machine files stay because they are the proof and the replay.
Everyone
README.md: The story of this run: who, what, why, where, when, the outcome, the decisions and the actions.triage/accounting-summary.json: The accounting rolled up: how many behaviours were reached, validated, handed to the team, or left as stated gaps, and why.assess/assess.md: The run against the blueprint's thirteen goals, each with the measurement and the file it came from.
Reviewers (the developer or team that owns the change)
intake/worklist.json: The work list: every package at base and head, what changed, what is reachable, what carries advisories.analyze/python-jose/facts.json: The fact bundle for one package: versions, API diff, call sites, advisories, the risk score and the budget it earned.analyze/python-jose/api-diff.json: Every added, removed or changed public symbol between the two versions, with a breaking flag.generate/python-jose/tests/: The candidate test files as generated, before any decision.execute/python-jose/results.json: The verdict per test: outcome on head, on the re-run and on the other version, coverage, and how each was judged.execute/python-jose/mutation/mutation.json: How strong the accepted tests are: which mutants of the package's source they killed, and the score.execute/python-jose/relevance.json: Which existing tests this change makes obsolete or redundant, proposed with evidence; nothing is deleted.triage/python-jose/triage.json: Every verdict and finding classified, with a confidence and a route; below the threshold a person decides.triage/python-jose/accounting.json: The accounting: every behaviour the change reaches in exactly one state, validated, for your environment or a stated gap with its reason, with the evidence per behaviour.packet/python-jose/packet.md: The review packet: the verdict in plain terms, one row per test, the findings, the VEX drafts, the retirements, the recommended action.packet/python-jose/tests.patch: The accepted tests as a patch in the layout they will live in.packet/python-jose/pull-request.md: The text and the file list the test pull request carries, whether or not one was opened.
Product Security
intake/vulns.json: Known vulnerabilities for every package version in either graph, from OSV, including malicious-package reports.analyze/python-jose/vulns.json: The advisories on this package's versions, with the symbols they name and the fixed versions.packet/python-jose/vex.openvex.json: One draft VEX statement per advisory, with its basis; a proposal for Product Security, never published by the harness.
Supply Chain Security and auditors
run.json: What was run, with which tool versions, on which repository by name.selfcheck/selfcheck.json: Stage 0: every failure-register check and the sandbox and model probes, before any evidence was produced.intake/sbom.new.cdx.json: The software bill of materials at head, CycloneDX.generate/python-jose/model-calls/: Every prompt and every response, numbered, with the call's settings; the full transcript of what the model saw and said.attest/python-jose/MANIFEST.json: The provenance record per accepted test: model, prompt and response digests, sandbox image digest, verdicts.attest/python-jose/statement.json: The in-toto test-result statement; its subjects are the patch, the record and each evidence record's head.attest/python-jose/statement.dsse.json: The statement signed as a DSSE envelope.attest/python-jose/signer.pub.pem: The public key that verifies the envelope.attest/python-jose/attest.json: Which key signed, whether it verified locally, and what the run could not back.attest/python-jose/udlm/: The same facts as sealed UDLM records: the vulnerability, the package, the run, each accepted test, the draft VEX.
The person who ran the harness
Tools (kept for replay and proof; not meant to be read)
run-index.json: What this run holds: stages, packages, headline numbers, and where every file is; the renderers read it.analyze/python-jose/api.new.json: The public API surface at head, extracted by tools.analyze/python-jose/api.old.json: The public API surface at base.analyze/python-jose/call-sites.json: Every place the application references this package.execute/python-jose/new/: The sealed run on head: the run script, the junit report, coverage, logs.execute/python-jose/new-rerun/: The second run on head, to catch flakes.execute/python-jose/old/: The run on the base version, for the differential.execute/python-jose/mutation/mutants/: Each sampled mutant: its change as a patch, and its sealed run's output.packet/python-jose/packet.json: The packet's facts in structured form.assess/assess.json: The assessment in structured form.