Prompting, Evaluation and Iteration
Choose the demonstrations Botie will see
BOTIE
Examples, experiments and workspace controls
Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.
Try your own inputs
Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.
Context for this challenge
Connect the example pool and requested kind to the development-example selector. Inspect the records before considering their size.
Botie has a pool of reviewed examples, but only part of it belongs in the prompt.
Inspect the component calculation
Choose development examples for this job( Example pool: Example records Requested kind: Requested example kind )
Examples and results
- Example records
[ { "id": "A", "kind": "manual", "split": "development", "tokens": 40, "input": "Question A", "output": "Reviewed answer A" }, { "id": "B", "kind": "manual", "split": "development", "tokens": 30, "input": "Question B", "output": "Reviewed answer B" } ]- Requested example kind
manual
- Required result
[ { "id": "A", "kind": "manual", "split": "development", "tokens": 40, "input": "Question A", "output": "Reviewed answer A" }, { "id": "B", "kind": "manual", "split": "development", "tokens": 30, "input": "Question B", "output": "Reviewed answer B" } ]
- Example records
[ { "id": "D", "kind": "manual", "split": "development", "tokens": 90, "input": "Question D", "output": "Reviewed answer D" }, { "id": "E", "kind": "manual", "split": "development", "tokens": 20, "input": "Question E", "output": "Reviewed answer E" }, { "id": "F", "kind": "booking", "split": "development", "tokens": 5, "input": "Question F", "output": "Reviewed answer F" } ]- Requested example kind
manual
- Required result
[ { "id": "D", "kind": "manual", "split": "development", "tokens": 90, "input": "Question D", "output": "Reviewed answer D" }, { "id": "E", "kind": "manual", "split": "development", "tokens": 20, "input": "Question E", "output": "Reviewed answer E" } ]
Try a fresh construction
This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.
Stuck? Inspect a worked construction
The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.
Assumptions and test precision
These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.
Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.
The relevant pool contains only a held-out case
This check uses the pieces from “Select before spending the context allowance”.
BOTIE
- Example records
[ { "id": "fresh-A", "kind": "booking", "split": "held-out", "tokens": 12, "input": "Can I book P12?", "output": "Approval needed." }, { "id": "fresh-B", "kind": "manual", "split": "development", "tokens": 8, "input": "P12 revision?", "output": "Ask for the label." } ]- Requested example kind
booking
- Tokens reserved for examples
100
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
Which contrasting example would you add if every selected answer recommends inspection?
Use examples to expose a task distinction, then test it on questions the prompt has not seen.
These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.
Work through the Python challenges →