Prompting, Evaluation and Iteration

Follow the technician’s request

The manual and equipment lookups run together

The maintenance assistant works across tools, but its answer cannot explain a slow request. Botie needs to follow the technician’s own execution through the recorded spans.

BOTIE

The request spans 0–900 ms, while child spans cover 50–350, 100–500 and 500–850 ms. Why would adding the child durations overstate this request’s wait?
The inputs we are comparing

Recorded span list

Lookups happen one after another

[
  {
    "id": "request",
    "parent": null,
    "start": 0,
    "end": 800,
    "trace": "pump-7"
  },
  {
    "id": "manual",
    "parent": "request",
    "start": 0,
    "end": 200,
    "trace": "pump-7"
  },
  {
    "id": "equipment",
    "parent": "request",
    "start": 200,
    "end": 500,
    "trace": "pump-7"
  },
  {
    "id": "answer",
    "parent": "request",
    "start": 500,
    "end": 800,
    "trace": "pump-7"
  }
]

Two lookups overlap

[
  {
    "id": "request",
    "parent": null,
    "start": 0,
    "end": 900,
    "trace": "pump-7"
  },
  {
    "id": "manual",
    "parent": "request",
    "start": 50,
    "end": 350,
    "trace": "pump-7"
  },
  {
    "id": "equipment",
    "parent": "request",
    "start": 100,
    "end": 500,
    "trace": "pump-7"
  },
  {
    "id": "answer",
    "parent": "request",
    "start": 500,
    "end": 850,
    "trace": "pump-7"
  }
]

Trace ID to inspect

Same in both cases

pump-7
Result for lookups happen one after another
800
Choose an explanation to test
The overnight export begins with someone else’s request

This check uses the pieces from “Measure the interval the technician waited”.

BOTIE

The export mixes pump-9 and pump-7. Measure only pump-7’s request duration, keeping the overlapping manual and equipment calls out of any sum.
Trace ID to inspect
pump-7
Recorded span list
[
  {
    "id": "other",
    "parent": null,
    "start": 0,
    "end": 5000,
    "trace": "pump-9"
  },
  {
    "id": "manual",
    "parent": "request",
    "start": 240,
    "end": 700,
    "trace": "pump-7"
  },
  {
    "id": "request",
    "parent": null,
    "start": 200,
    "end": 1000,
    "trace": "pump-7"
  },
  {
    "id": "equipment",
    "parent": "request",
    "start": 260,
    "end": 850,
    "trace": "pump-7"
  }
]

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Which connection prevents another technician’s root span from changing your answer, and why can sibling durations overlap?

Preserve request identity and causal context before aggregating timings. A trace is evidence of recorded operations, not a guarantee of answer quality.

These invented complete traces have a single root and a shared clock. Real exports can be sampled, incomplete or affected by clock skew.

Work through the Python challenges →
Continue the courseCan We Find the Run Behind This Score? →
← Which Examples Should We Put in the Prompt?