← All workshopsRead Checking the List a Reader Actually Sees

Recommendations and Ranking

Measure usefulness in the visible list

BOTIE

Add relevant hits and divide by the number shown.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Add relevant hits and divide by the number shown.

Botie must judge the list a reader actually received. The denominator should match the number of visible recommendations.

Count the relevant binary labels and divide by the number of displayed guides.

Examples and results
Visible binary relevance labels
  1. 01
  2. 10
  3. 21
Required result
0.6667
Visible binary relevance labels
  1. 01
  2. 11
Required result
1
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

Use a nonempty binary list at a fixed cutoff. This computes precision, not recall or NDCG. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

A longer recommendation list adds weak matches

This check uses the pieces from “Measure precision on the shown guide list”.

BOTIE

Botie shows five recommended guides. The relevance list is in displayed order, with 1 for useful and 0 for irrelevant. Calculate precision for the full five-guide list.
Visible binary relevance labels
  1. 01
  2. 10
  3. 20
  4. 31
  5. 40

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What additional labels would be needed to measure recall instead?

Evaluation must match the visible cutoff and the reader's task, not merely the scoring model.

Use a nonempty binary list at a fixed cutoff. This computes precision, not recall or NDCG.

Work through the Python challenges →
Continue the courseFinding Unusual Readings Without Letting Them Set the Reference →
← Compressing a Sparse Table of Preferences