← All workshopsRead SentencePiece

Natural Language Processing

Make the space visible

What kind of reconstruction does the marker support?

Botie should retain word-separation information while building text pieces. The supplied normalised input uses a visible marker to keep spaces in the piece stream.

BOTIE

Botie needs to retain spacing. Which reconstruction claim is supported by this workshop's already-normalised input?
The inputs we are comparing

Normalised text

Example 1

red cat

Example 2

blue dog

Marker-aware vocabulary

Example 1

{
  "▁red": 1,
  "▁cat": 2
}

Example 2

{
  "▁blue": 3,
  "▁dog": 4
}
Result for example 1
  1. 0▁red
  2. 1▁cat
Choose an explanation to test
A word repeats after a space

This check uses the pieces from “Let the vocabulary see space markers”.

BOTIE

For this already normalised text, prefix the first word with the visible whitespace marker and replace the internal space with that marker too. Return the longest matching vocabulary pieces, preserving both occurrences.
Normalised text
cat cat
Marker-aware vocabulary
{
  "▁cat": 12
}

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why is the beginning marker needed even though there was no leading space in the supplied text?

A tokenizer's normalisation and boundary conventions jointly determine the representation it can reconstruct.

This demonstrates explicit space markers on already-normalised input. Full SentencePiece training and normalisation contain additional behaviour.

Work through the Python challenges →
Continue the courseGreedy Coverage →
← The Unigram Language Model