← All workshopsRead The Pattern Language Models Use

Natural Language Processing

Keep the word, number and punctuation apart

BOTIE

Group the input into letter, digit and punctuation regions, keeping a leading space with its word. Convert those regions to IDs using the supplied table.
Ready to test
0 passed · 0 failed · 1 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Group the input into letter, digit and punctuation regions, keeping a leading space with its word. Convert those regions to IDs using the supplied table.

Botie will group letters, digits and punctuation before subword processing. A leading space stays attached to its word so the preprocessing rule preserves that signal.

Apply the supplied pattern rule to separate letter, digit and punctuation regions, then retrieve their region IDs.

Examples and results
Input text
 hello42!
Region IDs
{
  "42": 2,
  " hello": 1,
  "!": 3
}
Required result
  1. 01
  2. 12
  3. 23
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

A decimal and repeated exclamation marks

This check uses the pieces from “Create regions for a later subword procedure”.

BOTIE

Apply the supplied teaching pattern to the message. It keeps an optional leading space with a letter run, groups consecutive digits, and groups consecutive symbols. Return the region IDs in order.
Input text
 cost12.50!!
Region IDs
{
  "12": 2,
  "50": 4,
  " cost": 1,
  ".": 3,
  "!!": 5
}

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why are these regions not necessarily the final tokens of a language model?

Separate pre-tokenization boundaries from the later algorithm that divides each region into subwords.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseMaximum Matching →
← Moses Rules