← All workshopsRead Greedy Coverage

Natural Language Processing

Choose the piece that covers more new ground

A shorter spelling covers more new positions

Botie has room for another vocabulary entry. The useful gain is how many still-uncovered positions it adds, which may have little to do with the piece's spelling length.

BOTIE

Why should xy be preferred to lengthy in the second case?
The inputs we are comparing

Uncovered positions per candidate

Example 1

[
  [
    0,
    1
  ],
  [
    2,
    3,
    4,
    5
  ],
  [
    6
  ]
]

Example 2

[
  [
    0,
    1,
    2
  ],
  [
    3
  ],
  [
    4,
    5
  ]
]

Candidate pieces

Example 1

  1. 0longer
  2. 1ab
  3. 2x

Example 2

  1. 0xy
  2. 1lengthy
  3. 2z
Result for example 1
ab
Choose an explanation to test
The longest named candidate adds nothing new

This check uses the pieces from “Choose the candidate with the largest additional coverage”.

BOTIE

These lists contain only positions each candidate would newly cover. Choose the candidate with the largest remaining coverage and return its name. The spelling length of a candidate is irrelevant.
Uncovered positions per candidate
[
  [
    2,
    7,
    9
  ],
  [],
  [
    0,
    1,
    3,
    4
  ]
]
Candidate pieces
  1. 0xy
  2. 1longword
  3. 2z

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why must the remaining coverage sets be updated before selecting another entry?

Marginal gains depend on what has already been chosen; avoid counting the same improvement twice.

The marginal position sets are supplied. This is vocabulary selection by coverage, not longest-first tokenisation.

Work through the Python challenges →
Continue the courseMoving a Vocabulary →
← SentencePiece