Which Examples Should We Put in the Prompt?

Botie keeps making an inspection sound mandatory. Show what a supported answer looks like, then check that the examples teach the distinction we intended.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Showing a Task Inside the Input

BOTIE

The 2020 GPT-3 paper studied tasks presented through instructions and a small number of demonstrations in the model’s input. In that setting, adapting to the task did not require gradient updates during inference. This made the contents of the prompt part of how the task was specified.

For our maintenance desk, that means we can show a supported answer and a request for clarification side by side. It does not guarantee that a model will infer the intended rule. We still have to test what it does with unfamiliar questions.

Sources for this history

Language Models are Few-Shot Learners, 2020

Our Example Accidentally Teaches “Always Inspect”

We give Botie an example: a noisy pump, a manual excerpt, and an answer recommending inspection. The next technician asks what a warning light means. Botie recommends an inspection again. Perhaps it copied the most obvious pattern in the example.

An example shows more than the desired format. It also shows which evidence led to which decision. We need cases that make the distinction visible, including a question the supplied evidence cannot answer. Then we need questions that were never used to choose those examples.

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

Step 1 of 2

Give the decision a reason

These are abbreviated teaching examples, not complete maintenance advice.

The differences matter more than repeated phrasing.
Question and evidence	Desired behaviour	What it demonstrates
Meaning of a warning; manual explains it	Explain and cite the passage	Use available support
Whether inspection is due; revision missing	Ask for the applicable revision	Do not fill an evidence gap
Request to book; no approval	Prepare a proposal only	An answer is not authority to act

Meaning of a warning; manual explains it
Desired behaviour
Explain and cite the passage
What it demonstrates
Use available support
Whether inspection is due; revision missing
Desired behaviour
Ask for the applicable revision
What it demonstrates
Do not fill an evidence gap
Request to book; no approval
Desired behaviour
Prepare a proposal only
What it demonstrates
An answer is not authority to act

The differences matter more than repeated phrasing.

Step 2 of 2

Keep a question out of the selection pool

Examples A and B were written while developing the prompt. C is a held-out test.

Even when C fits more easily, using it as a demonstration would compromise this test’s independence.
ID	Split	Recorded cost
A	development	40 tokens
B	development	55 tokens
C	held-out	20 tokens

A
Split
development
Recorded cost
40 tokens
B
Split
development
Recorded cost
55 tokens
C
Split
held-out
Recorded cost
20 tokens

Even when C fits more easily, using it as a demonstration would compromise this test’s independence.

Selection comes before packing

Eligibility and space are separate decisions. Selecting first keeps held-out examples out even when they fit.

Choose examples without peeking at the exam

Select relevant development examples, then fit them into the reserved prompt space.

BOTIE

Botie has a pool of reviewed examples, but only part of it belongs in the prompt.

Choose an example

Example records

kind

split

tokens

input

output

kind

split

tokens

input

output

Requested example kind

Tokens reserved for examples

Selected example IDs and remaining allowance

{
  "selected": [
    "A",
    "B"
  ],
  "remaining": 10
}

Follow the operations

1. Example records

[
  {
    "id": "A",
    "kind": "manual",
    "split": "development",
    "tokens": 40,
    "input": "Question A",
    "output": "Reviewed answer A"
  },
  {
    "id": "B",
    "kind": "manual",
    "split": "development",
    "tokens": 30,
    "input": "Question B",
    "output": "Reviewed answer B"
  }
]

2. Requested example kind
```
manual
```

3. Choose development examples for this job

[
  {
    "id": "A",
    "kind": "manual",
    "split": "development",
    "tokens": 40,
    "input": "Question A",
    "output": "Reviewed answer A"
  },
  {
    "id": "B",
    "kind": "manual",
    "split": "development",
    "tokens": 30,
    "input": "Question B",
    "output": "Reviewed answer B"
  }
]

4. Tokens reserved for examples
```
80
```

5. Fit examples into the reserved space

{
  "selected": [
    "A",
    "B"
  ],
  "remaining": 10
}

6. Result

{
  "selected": [
    "A",
    "B"
  ],
  "remaining": 10
}

These are invented, recorded examples. The browser runs your construction without calling a language model. Passing them checks the surrounding program, not the quality of a live model.

1. Write the rule before choosing a demonstration

Our rule is to answer from the supplied approved evidence and ask for missing information when that evidence is insufficient. Start there. Then choose examples in which the evidence changes what a good answer can say.

If every example ends in inspection, the prompt leaves an important boundary unexplained. Include a supported explanation that needs no proposed action. Avoid real customer details; synthetic examples can teach the structure without exposing private records.

Examples illustrate the written instruction. They do not grant access or override the tool’s permission checks.

2. Keep input and response together

A demonstration has an input and a reviewed response. The input should include the evidence available at that moment. The response should show only claims that evidence supports. A polished answer without its source passage teaches a less useful relationship.

Use the message format your provider supports. A common representation alternates user and assistant messages after the system instructions. Keep the current user request last and label retrieved material as data.

Do not mix an example answer into the current user’s request as though the user supplied it.

Inspect the structure before adding more examples

Each dictionary is one message. The role tells the API how that message participates in the conversation.

Python

messages = [
    {"role": "system", "content": "Use the supplied manual. Ask if evidence is missing."},
    {"role": "user", "content": "P7: which revision applies? Evidence: none."},
    {"role": "assistant", "content": "Which revision is listed on P7?"},
]
print([message["role"] for message in messages])

Output

['system', 'user', 'assistant']

The example response belongs to an assistant turn. The programming challenge will append a separate current request.

3. Reserve space before selecting examples

The request also needs instructions, tool definitions, retrieved material and room for the output. Decide how much of the context allowance examples may use. Measure costs with the provider’s tokenizer or usage tools, including message overhead where applicable.

Our workshop supplies recorded costs for complete examples. It selects the requested kind from the development split, then scans in order and accepts each example that fits. This first-fit rule is easy to inspect. It does not find the most informative combination.

Example allowance: 80 tokens
A costs 40 → select A; 40 remain
B costs 55 → skip B; 40 remain

The recorded costs are teaching fixtures, not token counts calculated by the browser. A production request needs a full context check after assembly.

4. An example can be correct and still be a poor choice

Two nearly identical supported answers may use the space that would have shown a missing-evidence case. An example may also depend on information absent from its input, or contain an obsolete policy. Review content as well as length.

For a small task, begin with a few deliberately contrasting demonstrations. Compare their order and selection on development cases. If dynamic retrieval chooses demonstrations, test that selector too: a relevance score does not enforce privacy or keep held-out questions out of the pool.

Longer prompts are not automatically better. The evaluation must justify the added examples and their cost.

5. Leave the test answers outside the prompt

Keep development cases separate from the held-out set used to assess the chosen prompt. Near duplicates can leak the answer just as easily as exact copies. Split by the unit that matters, such as asset family or customer conversation, when related records would otherwise cross the boundary.

Once you inspect held-out failures and use them to change the prompt, those examples have become development evidence for the next revision. Retain them as regression tests and collect fresh cases for another independent assessment.

Record the example IDs, their order and the prompt version with the run. A changed demonstration is a changed experiment.

6. Test the distinction the example was meant to teach

Try a new warning with sufficient evidence, then remove the decisive passage. The expected behaviour should change. Also try a different asset whose manual uses different wording. This checks whether the candidate learned a useful task pattern rather than simply repeating our sample answer.

A test double can verify message assembly. Only runs against the intended model can provide evidence about its responses, and those responses may vary across repetitions.

The next lessons will keep this distinction between testing our program and evaluating the model visible.