Are We Releasing the Code We Actually Checked?

Botie adds a dependency and rebuilds the project. Bind the review and independent checks to the exact revision and artifact that will be released.

14 sections · 7 questions · 1 coding challenge

1 / 14

The arrow keys move between sections too.

A Build Needed a Traceable Origin

BOTIE

Software supply chains grew beyond a single source tree. Builds consume dependencies and configuration, then produce artifacts that may be copied through several systems. SLSA uses provenance to describe where an artifact came from and how it was produced.

For an agent-assisted change, that helps us ask a precise question: does the evidence describe this artifact? Provenance can establish origin under a trusted verification policy. It does not establish that the source code is correct or that the package has no vulnerabilities.

Sources for this history

The Green Check Belongs to Yesterday’s Build

Botie fixes the parser, then adds a package to tidy the implementation. The dashboard still shows a passing test run. But that run happened before the package was added. Which version are we about to release?

The same problem appears when we install an MCP server or accept a generated workflow. We need to know which code we reviewed, who checked it and whether the checked artifact is the one we are about to use.

These are invented records and deterministic policy checks. The browser runs no agent, shell command, MCP server or network request. Passing these fixtures establishes only the stated policy behaviour.

Step 1 of 1

Same green label, different evidence

The release candidate is revision r2, artifact a2.

Both identity and coverage matter.
Evidence	What it checked	Decision
Passing tests for r1/a1	An earlier build	Hold
Passing tests for r2/a2	The candidate	Inspect remaining requirements
Missing dependency result	No evidence for that requirement	Hold

Passing tests for r1/a1
What it checked
An earlier build
Decision
Hold
Passing tests for r2/a2
What it checked
The candidate
Decision
Inspect remaining requirements
Missing dependency result
What it checked
No evidence for that requirement
Decision
Hold

Both identity and coverage matter.

A check applies to the version it inspected

Neither matching identity nor passing checks can stand in for the other.

Bind the checks to the candidate

Compare artifact identity independently from the required test results.

BOTIE

The candidate is r2/a2. Some reports refer to an earlier artifact or omit a required result.

Choose an example

Current candidate

revision

artifact

Verified build evidence

revision

artifact

trustedRunner

True

checks

tests

dependencies

Required check names

Release review route

ready for release review

Follow the operations

1. Current candidate

{
  "revision": "r2",
  "artifact": "a2"
}

2. Verified build evidence

{
  "revision": "r2",
  "artifact": "a2",
  "trustedRunner": true,
  "checks": {
    "tests": "pass",
    "dependencies": "pass"
  }
}

3. Bind evidence to the artifact
```
true
```
4. Required check names
1. 0tests
2. 1dependencies
5. Check every required result
```
[]
```
6. Hold unverified releases
```
ready for release review
```
7. Result
```
ready for release review
```

1. Identify the new code before installing it

Check the package’s actual name, publisher, source and intended purpose. A plausible name in model output is not evidence that a package exists or belongs to the expected project. Watch for lookalike names and unexpected registry changes.

Review transitive dependencies and installation behaviour according to the task’s risk. For a local MCP server, installation means preparing code that may later run with host access. The server’s protocol support does not review its package for you.

A dependency is part of the program you will execute.

2. Pin inputs without mistaking a pin for trust

Record exact versions or immutable identifiers, preserve lockfiles and verify integrity through the package system’s supported mechanisms. A mutable tag can point somewhere different later. Approved input identity makes a change visible.

Pins can preserve vulnerable code too. Monitor advisories, review updates and retain a procedure for urgent replacement. The point is controlled change with evidence, not freezing every dependency forever.

Reproducibility and safety answer different questions.

3. Protect the build and its verifier

Run untrusted candidate code without release credentials. Keep protected checks and deployment policy outside the candidate’s editable scope. Use restricted job tokens and review workflow changes that would widen them.

An agent may generate a convincing CI result file. The release controller must obtain evidence from a trusted runner or verified attestation channel, not from a path the candidate can overwrite. Test output is evidence only within that trust relationship.

The fixture’s trustedRunner flag stands for that verified channel. A client must never be allowed to set it.

4. Bind evidence to an exact artifact

A revision identifies source state. An artifact digest identifies the bytes being released. Record both, together with the build inputs and required check results. Verify signed provenance against trusted issuers and identities where the pipeline uses it.

Our short labels make the relationship readable, but they are not hashes or signatures. Do not implement attestation verification by trusting a JSON field saying verified. Use maintained verification tooling and an explicit trust policy.

Changing the artifact after checking it requires fresh evidence.

A matching revision can still have a different artifact

Compare each identity independently.

Python

candidate = {"revision":"r8", "artifact":"a8"}
evidence = {"revision":"r8", "artifact":"a7"}
print(candidate["revision"] == evidence["revision"])
print(candidate["artifact"] == evidence["artifact"])

Output

True
False

The source label matches while the artifact label does not.

5. Decide which checks are mandatory

List required behaviour tests, dependency and secret checks, and any security checks specific to the changed boundary. Preserve pass, fail, skipped and unavailable as different states. The last two do not satisfy a mandatory passing requirement.

A scanner’s passing result means it found no issue under its database and configuration. Keep those versions with the result. Review application-specific risks that the scanner cannot establish, such as whether a write belongs to the correct tenant.

A count of green jobs is less useful than evidence for every declared requirement.

6. Review what an agent may promote

Allow the coding worker to submit a candidate with evidence. Keep release authority in a separately controlled process. Approvals should bind the reviewed revision, artifact and destination; a changed destination can matter as much as changed code.

Use small releases with a rollback or disable route appropriate to the effect. Reverting code does not undo a message already sent or data already disclosed. Preserve an action-reconciliation plan for those cases.

Release review decides whether this candidate may be exposed under stated limits.

7. Test the stale-evidence case deliberately

Give the controller a passing result for an earlier artifact. It should hold the new release even when every check label says pass. Then supply matching identity with one skipped check. That should hold for a different reason.

Keep these regression cases when the pipeline changes. The next lesson applies the same concern to a live MCP connection whose advertised tools can change after initial review.

The boundary must notice changes after the first successful review.