← All workshopsRead Maximum Matching

Natural Language Processing

Try the longest piece first

Two complete ways through abcd

Botie must segment text without spaces using a dictionary. Longest-first matching makes a local choice, so we need to see what it selects before treating that choice as the intended reading.

BOTIE

Both abc followed by d and ab followed by cd cover the text. Which path does longest-first matching select?
The inputs we are comparing

Text

Example 1

therapist

Example 2

abcd

Dictionary with IDs

Example 1

{
  "the": 1,
  "therapist": 2,
  "rap": 3,
  "ist": 4
}

Example 2

{
  "abc": 1,
  "ab": 2,
  "cd": 3,
  "d": 4
}
Result for example 1
  1. 02
Choose an explanation to test
A long match leaves a reusable shorter suffix

This check uses the pieces from “Take the longest dictionary entry at the current position”.

BOTIE

Tokenise from left to right by taking the longest dictionary match at each current position. Return the IDs, then check whether your construction resumes from the end of each chosen piece.
Text
abcab
Dictionary with IDs
{
  "abc": 8,
  "ab": 3,
  "c": 2,
  "a": 1,
  "b": 4
}

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What information would you need to prefer one complete segmentation beyond the length of its first piece?

A local longest match is a decoding rule, not evidence of linguistic correctness.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseThe Word Lattice →
← The Pattern Language Models Use