Where Did the Answer First Go Wrong?

Botie gives an incorrect inspection interval. We could add “be accurate” to the prompt, but first we should look at what it received. The lookup returned last year’s manual. The model might have repeated that passage faithfully.

Another run retrieves the current manual and still chooses the wrong interval. Those failures need different investigations. We will turn task failures into a review queue, identify the first observed divergence, and keep uncertain diagnoses open.

Keep failed requests, group the known locations, and preserve an unresolved queue.

{
  "counts": {
    "evidence": 2,
    "decision": 0,
    "execution": 0
  },
  "unresolved": []
}

Save the request, configuration versions and permitted outcome with the failed run. Preserve the tool observations that were actually supplied, subject to the application’s privacy and retention rules. Re-running the question against today’s database may produce different evidence.

Use an isolated fixture to reproduce the failure. If a write was involved, begin from a snapshot or fake service so reproduction does not repeat the unwanted effect.

A trace is an observation of one run. A replay needs an explicit environment if we want to compare it meaningfully.

For a wrong interval, check whether the requested asset was understood, whether the appropriate manual was returned, and whether the answer was supported by that manual. For a wrong work order, inspect the actual write and approval record as well.

The first observed divergence is a useful place to investigate. It is not automatically the sole cause: a second bug may be hidden because the first prevented the later step from being reached.

After repairing the earliest failure, rerun the complete task. The next failure may only then become visible.

Our teaching labels are evidence for an unsuitable or missing observation, decision for an incorrect choice despite sufficient evidence, and execution for a correctly specified call that the application handles incorrectly. An incomplete trace gets an unresolved label.

Real applications may need more precise categories. Define them with examples, allow reviewers to disagree, and record what evidence supported the decision. Do not let a model-generated category become unquestioned ground truth.

Keep critical violations visible even when they are rare. Frequency is not the same as severity.

Filter to failed tasks before counting stages. Successful runs do not belong in this failure inventory. A failed run with an unresolved stage still belongs in the inventory, but in a separate queue.

Use unique run IDs. If several attempts belong to one user request, decide whether this report counts attempts or whole tasks and keep that unit consistent. Our fixture has one reviewed run per task.

Failed tasks: A=evidence, B=decision, C=unresolved
Known counts: evidence=1, decision=1, execution=0
Unresolved queue: [C]

The counts describe this reviewed sample. They are not deployment failure rates.

reviews = [{"id":"A", "passed":False}, {"id":"B", "passed":True}]
failed_ids = [r["id"] for r in reviews if not r["passed"]]
print(failed_ids)

['A']

For the stale manual, propose: enforcing the current approved revision in retrieval will fix these interval questions without breaking requests for historical revisions. That names both the intended improvement and a possible regression.

Create a case that fails under the current retrieval rule. Add a historical-revision request whose legitimate answer should stay unchanged. A prompt instruction might reduce the error, but it cannot make an absent revision appear in the tool result.

A specific hypothesis tells us what to change and which neighbouring behaviour deserves protection.

Separate requests by meaningful task properties, such as missing asset IDs, old manual revisions or write permissions. Report sample sizes beside the results. A common easy task can conceal a severe failure in a small group.

Choose repair priorities using impact, occurrence and the quality of available evidence. Keep that decision explicit rather than pretending the largest bucket automatically deserves the next change.

Use the unresolved queue to improve instrumentation when it is preventing diagnosis. More prompt revisions will not recover a missing tool trace.

Run the proposed repair on the original failure and the neighbouring cases. Inspect the intermediate observation as well as the final answer. If the right manual now arrives but the answer stays wrong, the original retrieval issue was real and another issue remains.

Record that result. It is useful evidence, even though the first repair did not finish the task.

The following lesson compares a single declared change while preserving case identity and regressions.