Natural Language Processing
Give Botie a vocabulary
BOTIE
Examples, experiments and workspace controls
Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.
Try your own inputs
Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.
Context for this challenge
Build a word tokenizer for these messages: separate whitespace-delimited words, then use the vocabulary to turn each piece into an ID. Repeated words must keep their IDs.
Botie needs numerical addresses for the pieces of a readable message. Whole words provide a first vocabulary without treating those addresses as measurements of meaning.
Separate whitespace-delimited words and look up every occurrence in the supplied vocabulary, preserving order and repeats.
Examples and results
- Message
cat dog cat
- Vocabulary
{ "cat": 103, "dog": 207 }- Required result
- 0103
- 1207
- 2103
Try a fresh construction
This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.
Stuck? Inspect a worked construction
The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.
Assumptions and test precision
This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.
A vocabulary starts its IDs at zero
This check uses the pieces from “Turn word occurrences into vocabulary addresses”.
BOTIE
- Message
dog bird dog
- Vocabulary
{ "dog": 0, "bird": 9 }
Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.
Build a construction in the workspace above first. This case will test those pieces.
What would be wrong with interpreting the difference between the cat and dog IDs as a difference in meaning?
A token ID is an address; a later embedding lookup supplies its numerical features.
This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.
Work through the Python challenges →