← All workshopsRead Positional Encoding

Natural Language Processing

Tell two identical tokens apart

Move the token without changing its identity

Two occurrences share the same token embedding. Botie needs their positions to leave different information in the input.

BOTIE

The token remains [1, 2], but its position vector becomes [1, 0]. What combined vector reaches the model?
The inputs we are comparing

Token embedding

Same in both cases

  1. 01
  2. 12

Position vector

Example 1

  1. 00
  2. 11

Example 2

  1. 01
  2. 10
Result for example 1
  1. 01
  2. 13

For a list use brackets, such as [2, 4]. For a matrix put each row in brackets. Text answers can stay as text.

A position contribution can subtract

This check uses the pieces from “Keep identity and position in one vector”.

BOTIE

Botie has a three-coordinate token embedding and a supplied position vector containing a negative coordinate. Combine them as the model input. These are illustrative coordinates, not a full sinusoidal encoding.
Token embedding
  1. 02
  2. 1-1
  3. 20.5
Position vector
  1. 0-1
  2. 10.5
  3. 20

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What information disappears if the position branch is disconnected?

These supplied vectors show additive position information; they do not calculate a sinusoidal or rotary encoding.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseAttention →
← GloVe