What Happens When the New Shift Finds a Failure?

Botie’s next version passes the offline tests and starts handling a small part of the shift. A technician reports a job she never approved. We can switch back to yesterday’s prompt, but a worker may already have received the new version’s queued request.

Now we need to know what actually changed, stop further unwanted work and account for the requests already moving through the system. We will build a small release control and an action inventory, then work through the decisions that a real incident procedure must cover.

Check the selected release’s current evidence and stop control, then keep a separate inventory of its unsettled actions.

[
  "continue_canary",
  [
    "job-A"
  ]
]

A maintenance request should record the release that handled it, including the code and relevant configuration revisions. The same prompt can behave differently after a tool permission or retrieval source changes. Keep those changes discoverable.

Choose a named owner who can pause the rollout and an escalation route when that person is unavailable. A stop rule nobody can enact is only a description.

The workshop selects one release by ID. The real service must attach that identity when work is created, including work placed on a queue.

In shadow mode, the candidate receives an approved copy of the request stream while its outputs stay out of the user’s workflow. Disable writes at the executor or route them to isolated test resources. Merely hiding the final answer does not stop a tool from creating a job.

A shadow request can still expose data to a provider or consume resources. Apply the same data-access and retention decisions as for other processing. Use synthetic replay if the real records cannot be copied appropriately.

Shadow results help compare decisions on realistic inputs. They do not test every consequence of serving live users.

A canary sends a deliberately limited portion of live work to the candidate while retaining the current version for comparison. Choose which tasks are eligible and how long the observation window lasts before starting.

Our example canary begins with authorised manual lookups. Expanding to proposed jobs requires its own evidence. A short quiet window may contain none of the cases that caused trouble offline, so traffic volume alone is a poor readiness measure.

The workshop’s continue_canary result permits only the already planned limited stage. It never means “roll out everywhere.”

An unapproved write is an immediate stop signal in this example. Other signals may include elevated tool errors, repeated budget exhaustion, missing telemetry or a rising false-refusal rate. Give each signal an observation window, threshold and owner.

Keep the candidate’s requests distinguishable from the baseline. Check rates with their counts, and check critical events individually. A mean quality score cannot compensate for a recorded unauthorised action.

An alert should point to the affected release and the evidence needed to investigate. Avoid copying full private prompts into notifications.

The small control we build requires a nonempty set of completed clear checks for the chosen release, fresh monitoring evidence and no operator stop request. If any condition fails, its result is pause_and_reconcile.

A second branch lists that release’s unsettled actions. Keep the inventory available even while the canary is healthy; it becomes useful immediately if the stop condition changes. An empty list means no unsettled actions were found in the supplied records, not that the entire external system has been audited.

Continue this limited stage = completed clear checks AND fresh monitoring AND no operator stop
Otherwise = pause_and_reconcile

This is an illustrative operating policy. It omits traffic allocation, persistence and enforcement in the actual tool gateway.

The application can stop accepting new autonomous work and revoke the affected release’s write authority at the tool gateway. Workers should check the current authority before executing queued work, so a request accepted earlier does not bypass the stop. Protect and audit the operator control itself.

Stopping a request may be best effort once a remote side effect has begun. Verify that workers received the change and inspect the target system. Do not report “cancelled” solely because the web interface stopped waiting.

A model should not be able to disable its own stop control or replace the operator’s decision with a generated explanation.

Use stable action IDs to connect the proposal, approval, dispatch and external result. Pending work can be held where the queue supports it. An in-flight or unknown result needs a status lookup before any retry.

A completed action may require a separate, authorised correction. Keep it in the incident audit even though our unsettled-actions function excludes it. A reversal can have its own consequences, and some effects cannot be undone.

The practice function creates an investigation list. It does not perform cancellation, retry or compensation.

actions = [
    {"id": "job-A", "status": "completed"},
    {"id": "job-B", "status": "unknown"},
    {"id": "job-C", "status": "pending"},
]
for item in actions:
    if item["status"] in {"pending", "in_flight", "unknown"}:
        print(item["id"], item["status"])

job-B unknown
job-C pending

Record when the event was detected, the affected release, the capability paused and who owns the next action. Link restricted evidence using request and action IDs. A short shared status record can explain what is known without reproducing private conversations.

Retain the minimum evidence needed to establish the sequence and impact. Apply access controls and retention to that evidence, and preserve its integrity. Redaction should leave the relationships needed for diagnosis; a broad log dump can create another exposure.

If notifications to affected users or organisations are required, assign that work to an owner. The model should not invent impact claims before the evidence is reconciled.

Write down how to enact the stop, verify it reached workers, locate unsettled actions and find the current on-call owner. Include the known-good fallback and the conditions under which it is appropriate. A fallback that shares the broken authorisation layer may repeat the same failure.

Practise with a synthetic incident: a queue contains one pending action, one unknown result and an old completed write. Ask another operator to follow the steps and note which identities or controls are missing. Update the procedure from that exercise.

The runbook should name actual controls and records in the deployed system. This lesson supplies the questions to answer, rather than pretending every service has the same commands.

First explain which control failed. Was approval bound to the wrong proposal? Did the executor trust a caller-supplied organisation? Did a retry create another effect? A new prompt may change the symptom while leaving the application boundary open.

Create a sanitised replay that fails for the original reason. Repair the responsible control, rerun prior checks and add fresh variants. Reconcile unsettled effects and have the release owner approve the next limited stage after inspecting the evidence.

Recovery has separate requirements for the repaired software and the already affected work. Clearing an alert does not establish either one.

A provider update, changed manual source or new class of technician request can change behaviour without a prompt edit. Continue comparing outcomes by relevant task group and configuration. A growing review queue or unknown-action count is a reason to investigate the system, even if average answer quality stays stable.

The next workshop shift includes a stale monitor and an unknown action. Predict which branch should react before running it. The goal is to understand why the machine pauses and what evidence someone needs next.

The same construction must handle an explicit stop and an unresolved check. Neither condition should be concealed by yesterday’s successful examples.