Repair Botie’s Maintenance Assistant

Follow a stale-manual failure through a test, a repair and a fresh evaluation. Keep unsupported answers and unapproved actions out of the release decision.

14 sections · 4 questions · 1 coding challenge

1 / 14

The arrow keys move between sections too.

Keeping a Repair From Becoming a New Failure

BOTIE

Behavioural tests make a discovered mistake repeatable. Experiment datasets then let a team run candidate versions against stored tasks and expectations. Langfuse supports datasets for this kind of comparison; the useful practice is retaining the case, configuration and observed result together.

Our project follows that practice with small visible records. The tools help us organise evidence, but the expected outcome still has to come from the task. A stored score with no clear acceptance rule is difficult to use when the assistant changes.

Sources for this history

Yesterday’s Fix Meets Today’s Shift

Our new prompt handles the missing-revision question. The next shift brings a different failure: the manual lookup finds an old revision, and Botie treats it as current. We have reached the point where adding another sentence to the prompt is no longer the obvious repair.

We will work from the actual recorded failure, change the component responsible, and preserve the behaviours that already worked. The final decision needs evidence from a fresh shift, including a case that does not return a result.

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

Step 1 of 2

Carry the failure into the next experiment

Each revision responds to observed evidence, with a neighbouring behaviour protected.

These are separate revisions. Changing all of them together would make diagnosis harder.
Observation	Candidate change	Retained check
Missing revision produces a guess	Contrast supported and unsupported examples	Still answer supported questions
Old manual returned as current	Repair revision filtering in the tool	Allow explicit historical questions
A fresh evaluation times out	Repair the harness or dependency	Do not remove the missing case

Missing revision produces a guess
Candidate change
Contrast supported and unsupported examples
Retained check
Still answer supported questions
Old manual returned as current
Candidate change
Repair revision filtering in the tool
Retained check
Allow explicit historical questions
A fresh evaluation times out
Candidate change
Repair the harness or dependency
Retained check
Do not remove the missing case

These are separate revisions. Changing all of them together would make diagnosis harder.

Step 2 of 2

One case travels through the entire repair

Keep its identity as it moves from an observed mistake to a retained test.

1
Failure
- Wrong interval for P7
- Inspect the trace
2
Repair
- Current-revision filter
- Rerun original and neighbouring cases
3
Assessment
- Freeze candidate
- Evaluate unseen shift requests

A limited trial still needs monitoring and a way to stop or roll back.

The missing run belongs in the decision

Coverage and acceptance remain separate. An absent observation cannot disappear before the acceptance check.

Does the repair have enough evidence for a limited trial?

Preserve every required case, then check task results, critical effects and latency.

BOTIE

The old failure passes. Botie now needs the whole candidate report, including the request whose worker did not return.

Choose an example

Required case IDs

Candidate measurements

passed

True

critical

False

passed

True

critical

False

Maximum milliseconds per case

Trial eligibility and reasons to hold

{
  "decision": "eligible for limited trial",
  "reasons": []
}

Follow the operations

1. Required case IDs
1. 0current
2. 1missing

2. Candidate measurements

[
  {
    "id": "current",
    "passed": true,
    "critical": false,
    "ms": 400
  },
  {
    "id": "missing",
    "passed": true,
    "critical": false,
    "ms": 700
  }
]

3. Keep every required case in the report

[
  {
    "id": "current",
    "passed": true,
    "critical": false,
    "ms": 400
  },
  {
    "id": "missing",
    "passed": true,
    "critical": false,
    "ms": 700
  }
]

4. Maximum milliseconds per case
```
1000
```

5. Check the candidate before a limited trial

{
  "decision": "eligible for limited trial",
  "reasons": []
}

6. Result

{
  "decision": "eligible for limited trial",
  "reasons": []
}

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

1. Recover the evidence for the wrong interval

The technician asked for the current P7 interval. The trace records a tool result from revision 2 while the approved current revision is 4. The answer repeats revision 2. Save this as a development fixture with the requested time and expected revision.

Check the authoritative manual catalogue rather than treating a model judge as the source of revision truth. If the catalogue itself is ambiguous, resolve that uncertainty before writing an exact expected answer.

The first repair target is evidence selection. We still need to check the answer after the correct evidence arrives.

2. Make the tool repair small enough to inspect

Change the current-manual lookup to require the approved revision valid for the request. Preserve a separate explicit historical lookup when the task allows it. The tool contract should make the requested mode visible instead of silently choosing whichever record ranks highest.

Test that contract with a fake catalogue containing both revisions. Then run an integration test against an isolated instance of the actual service. A prompt saying “use the latest” cannot enforce a filter the tool does not implement.

Do not let the repaired read broaden access. The same caller and asset permissions still apply.

3. Retain useful prompt behaviour

The missing-evidence examples still matter. If the current manual is absent, Botie should request the needed information or hand off. It should not use the historical record merely because the tool happened to return one.

Keep a supported-answer case and an explicit historical-question case beside the original failure. These protect useful work while we tighten the current-record rule.

A repair that refuses every question can pass a narrow “no unsupported answer” check while making the assistant unusable.

4. Freeze the candidate before the fresh shift

Record the model, prompt, demonstration IDs, tool revision and dataset version. Repeat variable cases as needed under a documented policy. Keep critical permission tests alongside answer-quality checks, but report their outcomes separately.

Select fresh requests that vary asset names and evidence conditions. If we inspect their failures and revise again, they join the development record. Another independent assessment then needs new cases.

A fresh set should represent the intended users and tasks. Renaming P7 to P8 alone does not provide broad coverage.

5. Build a gate that cannot forget a required case

The workshop supplies required case IDs and candidate measurements. Join the measurements onto those IDs first. A missing result is a reason to hold, as is a failed task, an observed critical violation or latency above the declared limit.

Our example requires every listed case to pass. A real release policy may allow different quality thresholds for noncritical tasks, but it should be explicit before the candidate is evaluated. Critical effects should not be averaged away.

Required: supported, missing-evidence, approval
Observed: supported passed; missing-evidence passed
Missing: approval
Decision: hold

No required cases also means hold. An empty report cannot justify trial eligibility.

Find the case that never returned

Use set membership for coverage, but preserve the required order when presenting the missing IDs.

Python

required = ["supported", "missing-evidence", "approval"]
observed = {"supported", "missing-evidence"}
missing = [case_id for case_id in required if case_id not in observed]
print(missing)

Output

['approval']

The approval case remains required even though the worker returned no measurement.

6. Interpret eligibility without promising certainty

Passing this fixture means eligible for limited trial under its declared checks. It is not a claim that the assistant is safe for every request or ready for unrestricted writes. Define the trial population, approval boundaries, monitoring and stop conditions.

Watch real task failures and false refusals. Reconcile any ambiguous writes before retrying them. A rollback can change future requests without undoing effects already committed.

The release and incident-response lessons supply the operational checks that surround this decision.

7. Keep the next failure useful

When a trial request fails, retain its relevant evidence under the data policy and put it through the same diagnosis process. Ask which assumption broke and which part should change. Predict the effect before running a new candidate.

The aim is a sequence of explainable repairs. A prompt file with a growing pile of warnings is difficult to maintain when nobody remembers which failure each sentence was meant to prevent.

You now have a path from a user’s task to an observed failure, a testable hypothesis and a measured revision. Use it whenever the task or tools change.