Natural Language Processing

Count what actually goes on the wire

One character with several encoded bytes

Botie is counting storage for text. A visible character can require several bytes, so counting characters will not answer the transmission question.

BOTIE

How many UTF-8 bytes are needed for the supplied character 猫?
The inputs we are comparing

Message

Example 1

é

Example 2

猫
Result for example 1
2

For a list use brackets, such as [2, 4]. For a matrix put each row in brackets. Text answers can stay as text.

An accent is stored as a separate code point

This check uses the pieces from “Count the UTF-8 representation actually produced”.

BOTIE

This displayed é is stored as an ASCII e followed by a combining acute accent, U+0301. Return its UTF-8 byte count. Do not normalise the input before encoding it.
Message
é

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why do the supplied strings 猫 and cat have the same byte count despite different character counts?

Specify whether a text length counts bytes, code points, pieces or perceived symbols.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseSplitting on Spaces →
← What a Token Is