← All workshopsRead Moses Rules

Natural Language Processing

Separate the punctuation before translation

BOTIE

Separate punctuation before vocabulary lookup so the same word entry works beside different punctuation marks.
Ready to test
0 passed · 0 failed · 1 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Separate punctuation before vocabulary lookup so the same word entry works beside different punctuation marks.

Botie's translation input contains words beside punctuation. Separating those marks lets a word keep the same vocabulary entry across sentence contexts.

Separate punctuation from adjacent words, preserve all pieces in order, and map them through the supplied vocabulary.

Examples and results
Translation input
red,blue
Word and punctuation IDs
{
  "red": 1,
  "blue": 2,
  ",": 3,
  "!": 4
}
Required result
  1. 01
  2. 13
  3. 22
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

This constructs the punctuation-separation step. Full Moses tokenisation includes additional language-specific rules and escaping conventions. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Repeated punctuation in a translation message

This check uses the pieces from “Present words and punctuation consistently to translation”.

BOTIE

Apply the workshop's teaching rule: group letters and numbers, but keep each punctuation character as a separate piece. Return the vocabulary IDs for the full message, including every dot.
Translation input
Wait...really?
Word and punctuation IDs
{
  "Wait": 1,
  ".": 2,
  "really": 3,
  "?": 4
}

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why should punctuation be preserved rather than discarded when preparing translation input?

This punctuation step is only part of Moses-style preprocessing, which also has language-specific conventions.

This constructs the punctuation-separation step. Full Moses tokenisation includes additional language-specific rules and escaping conventions.

Work through the Python challenges →
Continue the courseThe Pattern Language Models Use →
← Penn Treebank Rules