Notice the punctuation in another script

A different comma between familiar letters

Botie receives punctuation from more than one writing system. Punctuation must remain a visible piece even when it is not an English comma.

BOTIE

The text red،blue uses an Arabic comma. What should change relative to splitting around another recognised punctuation mark?

The inputs we are comparing

Text

Example 1

猫，狗

Example 2

red،blue

Vocabulary

Example 1

{
  "猫": 1,
  "，": 2,
  "狗": 3
}

Example 2

{
  "red": 4,
  "،": 5,
  "blue": 6
}

Result for example 1

Choose an explanation to test

The punctuation piece changes, while the separation rule still appliesThe whole string must stay one word because there is no ASCII comma

Accented letters beside East Asian punctuation

This check uses the pieces from “Keep cross-script punctuation as its own piece”.

BOTIE

Use the workshop's Unicode-aware rule to group consecutive letters and numbers while preserving individual punctuation marks. Return IDs for the resulting pieces from the supplied vocabulary.

Text: café、茶!
Vocabulary: { "café": 21, "、": 7, "茶": 8, "!": 2 }

What result should your machine produce for this case?

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why does correctly separating punctuation not solve word segmentation in every script?

Your notes for later. These stay with this journey.

Punctuation recognition is one boundary operation, not a complete linguistic segmentation method.

This small Unicode-aware rule illustrates punctuation handling. It does not implement the complete Unicode word-boundary standard or linguistic word segmentation.

Work through the Python challenges →

Continue the coursePenn Treebank Rules →

← Splitting on Spaces