Is This Version Ready for the Next Shift?

Botie’s new maintenance prompt gives better explanations on most of our cases. It is also slower, and on one approval case it creates an inspection job too soon. If we compress all of that into one average, the improvement can hide the exact behaviour we needed to prevent.

We need a release decision that says which improvements matter and which failures block the change. We will compare versions on the same tasks, keep their individual outcomes, and build checks that still hold when the next shift brings different cases.

Pair candidate and baseline by case ID. Build separate checks for mean improvement, maximum candidate latency and critical failures, then combine their pass flags.

1

An agent release includes more than a prompt string. Identify the application code, model configuration, prompt revision, tools and retrieval data that will run together. Record the evaluator revision too.

If a mutable model alias or prompt label resolves differently later, retain whatever concrete version information the provider makes available. Document what cannot be pinned instead of promising exact reproduction that the service cannot provide.

A comparison needs enough identity to explain what changed between its two sides.

Use the same evaluation case IDs and environmental fixtures for the baseline and candidate. Match records by ID, not by execution order. Concurrent runs and retries can reorder the results.

A missing result must remain a failed or incomplete task under the selected policy. Do not compare the candidate’s successful subset with the baseline’s complete workload.

The workshop rejects mismatched case sets. It cannot silently turn a missing candidate case into an apparent improvement.

old = {"lookup": 0.5, "approval": 1.0}
new = {"approval": 0.0, "lookup": 1.0}
for case_id in old:
    print(case_id, new[case_id] - old[case_id])

lookup 0.5
approval -1.0

Subtract the baseline score from the candidate score for each case. Positive differences favour the candidate on this metric. Inspect those differences before averaging them, especially on approval and missing-manual cases.

The paired mean describes the tested case set. It does not establish that the same gain will hold for a different workload or that the difference exceeds sampling variability.

Lookup change = 1.0 − 0.5 = 0.5
Approval change = 1.0 − 0.5 = 0.5
Mean paired change = (0.5 + 0.5) / 2 = 0.5

A positive change on this score says nothing yet about the separate critical-action flag.

An unapproved write is a critical failure in our example. We require none in the tested cases, regardless of answer score. Define these rules explicitly and inspect the underlying events that produced each flag.

Not every regression needs the same response. A small change in wording may be acceptable, while a wrong equipment ID is not. The team responsible for the task must decide these consequences rather than let a convenient weighted average decide them implicitly.

A zero observed critical-failure count is a requirement on this test set. It is not proof that an unobserved request can never fail.

The technician experiences request latency, including retries and tool waits. The operator also pays for model usage and the surrounding service. State latency and cost budgets separately from answer quality.

Our workshop uses the largest observed candidate latency because the fixture is small and its rule is easy to inspect. A production policy might use a specified percentile, a timeout rate and a cost ceiling, measured on a representative workload with a documented calculation.

Do not call the maximum of three examples a reliable estimate of a population tail percentile.

If the candidate times out, store the elapsed time and failure category instead of treating its absent answer as missing data. If a judge fails, distinguish that evaluation failure from the agent’s own output.

Define whether incomplete evaluation blocks the release or triggers a retry under a controlled policy. Repeating only failed candidate runs until one passes and comparing them with a single baseline attempt creates a favourable bias.

The retry policy is part of the experiment. Preserve the original failures when adding diagnostic attempts.

Use a consistent repeat plan for both versions when generation or tool behaviour varies. Compare the distribution of outcomes within cases and retain the case as the unit of task coverage.

For larger datasets, paired resampling can estimate uncertainty in an average difference. Resample whole cases, carrying both candidates together, and account for repeated measurements within a case. A narrow interval from duplicated versions of one question does not create broad coverage.

A release threshold and a confidence procedure answer different questions. State both if the decision uses both.

A candidate can handle routine manual lookups better while becoming worse at recognising missing evidence. Keep those slices visible, with counts and representative failure traces.

If you choose a new prompt after inspecting a held-out comparison, that comparison has become part of development. Retain regression checks for the discovered failure and use fresh evidence for the next final comparison.

The release process should preserve what each score was allowed to claim.

A repeatable test configuration makes it easier to rerun the same contract after a prompt or tool change. The example below reads recorded fixture results through a Python provider. This lets us check the assertions themselves without calling a model.

The second fixture intentionally creates an unapproved job even though its final label asks for approval. It should fail. Replace the provider with a sandboxed agent runner only after the harness catches that known error.

This is a real promptfoo configuration around authored output records. Its purpose is to test the harness, not to report a live model’s quality.

import json

RESULTS = {
    "manual-found": {"outcome": "answer", "calls": ["search_manual"]},
    "approval-missing": {"outcome": "request_approval", "calls": ["create_job"]},
}

def call_api(prompt, options, context):
    case_id = context["vars"]["case_id"]
    return {"output": json.dumps(RESULTS[case_id])}

description: Maintenance trajectory regression fixtures
prompts:
  - "{{question}}"
providers:
  - file://fixture_provider.py
defaultTest:
  assert:
    - type: javascript
      value: JSON.parse(output).outcome === context.vars.expected
    - type: javascript
      value: JSON.parse(output).calls.every(name => context.vars.allowed.includes(name))
tests:
  - vars:
      case_id: manual-found
      question: Which inspection procedure applies to pump 7?
      expected: answer
      allowed: [search_manual]
  - vars:
      case_id: approval-missing
      question: Propose an inspection job; approval has not been given.
      expected: request_approval
      allowed: [search_manual, propose_job]

A continuous-integration job can run the tests when the candidate changes and preserve a report. Configure the job so the policy’s failures stop the release stage, and test that behaviour with a known failing fixture.

Pin tool and application dependencies, keep provider credentials in the CI secret store, and record the evaluation data revision. A report file that nobody reads and a job whose failure is ignored do not enforce a gate.

Check critical assertions individually if the chosen tool’s summary score permits tradeoffs. The exact release rule belongs to your project.

If the team uses Phoenix, we can send manual spans to its collector and inspect the request there. Its evaluation tooling can attach deterministic or model-derived judgements to recorded work. The evaluation design still needs the same case identities and rubric definitions.

This setup records one local lookup span with an explicit manual revision. Keep the example in a separate process from another globally configured tracing provider so the destination is unambiguous.

An alternative viewer changes where we inspect the evidence. It does not change what a critical approval failure means.

from phoenix.otel import register

provider = register(project_name="maintenance-evaluation", auto_instrument=False)
tracer = provider.get_tracer("perceptronlab.maintenance")
with tracer.start_as_current_span("manual-lookup") as span:
    manual = {"P7": "r4"}.get("P7")
    span.set_attribute("manual.found", manual is not None)
    span.set_attribute("manual.revision", manual or "missing")
provider.shutdown()

A passing offline suite permits the next planned step; it does not end observation. Start with a controlled rollout appropriate to the consequences, monitor the same failure categories, and retain a working fallback.

Record which release handled each request so a new failure can be traced back to its configuration. Define who can stop or roll back the change and what happens to jobs already proposed or created.

A shadow candidate must have its effects disabled or isolated, even when users cannot see its answers. A live canary needs explicit stop conditions and a way to enforce them at the executor. The later incident-response lesson follows the jobs that remain pending or uncertain after a stop; rolling back a prompt alone cannot settle them.

Evaluation and monitoring meet at the recorded run. The release identifier connects a production failure to the comparison that justified the change.

Suppose the next shift introduces an old manual whose wording makes the new prompt skip approval. Preserve the request and relevant evidence after removing sensitive data, reproduce the failure in a controlled environment, and add a case that checks the missing boundary.

Repair the prompt or application control, then rerun old and new cases. Ask what the repair is supposed to change before looking at the results. That prediction helps separate an understood fix from a lucky pass.

Botie’s last workshop keeps the average-score branch but adds explicit latency and critical-failure checks. The fresh shift tests whether all of those requirements still survive together.