← All workshopsRead Can We Make Botie Cross the Boundary?

Prompting, Evaluation and Iteration

Count the attack attempts without losing ordinary work

Botie refuses two ordinary maintenance requests

The approval guardrail stopped the recorded attacks, but technicians now struggle to get permitted work done. Botie needs separate evidence about the boundary and the service it is meant to protect.

BOTIE

Both approval attacks are prevented. Among three permitted requests, two are unjustifiably refused. Which branch tells us that the new defence harms the intended task?
The inputs we are comparing

Attack records: family and success (0, 1 or null)

The approval rule preserves useful requests

[
  {
    "family": "approval",
    "success": 0
  },
  {
    "family": "approval",
    "success": 0
  },
  {
    "family": "privacy",
    "success": 1
  }
]

A stronger refusal rule also blocks technicians

[
  {
    "family": "approval",
    "success": 0
  },
  {
    "family": "approval",
    "success": 0
  }
]

Attack family to inspect

Same in both cases

approval

Permitted requests: unjustified blocked flag

The approval rule preserves useful requests

[
  {
    "blocked": 0
  },
  {
    "blocked": 0
  }
]

A stronger refusal rule also blocks technicians

[
  {
    "blocked": 1
  },
  {
    "blocked": 0
  },
  {
    "blocked": 1
  }
]
Result for the approval rule preserves useful requests
[
  {
    "events": 0,
    "reviewed": 2,
    "total": 2,
    "rate": 0,
    "coverage": 1
  },
  {
    "events": 0,
    "reviewed": 2,
    "total": 2,
    "rate": 0,
    "coverage": 1
  }
]
Choose an explanation to test
The privacy family fails while the approval family looks clear

This check uses the pieces from “Keep permitted work beside the attack result”.

BOTIE

Select privacy. Two of its four reviewed attacks expose restricted data, and one additional attempt remains unresolved. Ordinary tasks have one false refusal among four reviewed requests. Keep both reports and their coverage.
Attack family to inspect
privacy
Attack records: family and success (0, 1 or null)
[
  {
    "family": "approval",
    "success": 0
  },
  {
    "family": "approval",
    "success": 0
  },
  {
    "family": "privacy",
    "success": 1
  },
  {
    "family": "privacy",
    "success": null
  },
  {
    "family": "privacy",
    "success": 0
  },
  {
    "family": "privacy",
    "success": 1
  },
  {
    "family": "privacy",
    "success": 0
  }
]
Permitted requests: unjustified blocked flag
[
  {
    "blocked": 0
  },
  {
    "blocked": 1
  },
  {
    "blocked": 0
  },
  {
    "blocked": 0
  }
]

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What would make a zero observed violation rate uninformative, and how would you check a repair without rewarding a system that refuses everything?

Retain the boundary, attack family and evidence coverage with every rate. Preserve useful permitted work and use fresh scenarios after a repair.

The fixtures contain authored outcome labels, not executed adversarial tests. A real evaluator must establish these labels from the isolated application’s trajectory and effects.

Work through the Python challenges →
Continue the courseWhat Happens When the New Shift Finds a Failure? →
← Is This Version Ready for the Next Shift?