← All workshopsRead Who Checks the Model That Checks Botie?

Prompting, Evaluation and Iteration

Give disputed answers a place to go

BOTIE

Compare the human and judge fields in each record, flagging either uncertainty or unequal labels.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Compare the human and judge fields in each record, flagging either uncertainty or unequal labels.

Botie needs a review queue for recorded answer judgements, including disagreements hidden by a favourable average.

Inspect the component calculation
Flag disputed judgements(
  Records with judgement labels: Answer judgement records: human, judge, reversed
  First label field: human
  Second label field: judge
)
Examples and results
Answer judgement records: human, judge, reversed
[
  {
    "id": "condition-kept",
    "human": 1,
    "judge": 1,
    "reversed": 1
  },
  {
    "id": "condition-lost",
    "human": 0,
    "judge": 1,
    "reversed": 1
  }
]
Required result
  1. 00
  2. 11
Answer judgement records: human, judge, reversed
[
  {
    "id": "short-answer",
    "human": 1,
    "judge": 1,
    "reversed": 0
  },
  {
    "id": "clear-error",
    "human": 0,
    "judge": 0,
    "reversed": 0
  }
]
Required result
  1. 00
  2. 10
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

These recorded labels are fixtures. Agreement is not proof of correctness, and a changed judgement does not by itself prove a specific source of bias. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.

The calibration batch agrees on both acceptance and rejection

This check uses the pieces from “Keep every reason for review attached to the case”.

BOTIE

Every recorded label is settled and stable in this batch. Return the review queue using the same rule, even when the correct queue is empty.
Answer judgement records: human, judge, reversed
[
  {
    "id": "restart-condition-kept",
    "human": 1,
    "judge": 1,
    "reversed": 1
  },
  {
    "id": "restart-condition-lost",
    "human": 0,
    "judge": 0,
    "reversed": 0
  },
  {
    "id": "wrong-equipment-advice",
    "human": 0,
    "judge": 0,
    "reversed": 0
  }
]

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why does the queue flag a problem without supplying a corrected verdict, and what can still be wrong when every available label agrees?

Use model judgements as fallible evidence. Preserve uncertainty and inspect the evaluator as well as the agent.

These recorded labels are fixtures. Agreement is not proof of correctness, and a changed judgement does not by itself prove a specific source of bias.

Work through the Python challenges →
Continue the courseWhere Did the Answer First Go Wrong? →
← What Can We Test Without Calling a Model?