← All workshopsRead Attention

Natural Language Processing

Let the query choose what contributes

BOTIE

Produce the three attention weights in the / red / door order.
Ready to test
0 passed · 0 failed · 1 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Set both query coordinates to zero and inspect the attention shares. Then change just one source word's value. Can you explain its influence on the answer from the share that word receives? Keep the key dimension consistent with the vectors.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Produce the three attention weights in the / red / door order.

Let's return to translation. While producing one target word, the model may need one part of the source sentence; at the next step it may need another. Our source is 'the red door'. The key rows and value entries follow that word order. We will keep them fixed and change the query to see how the contribution from each source word changes. These invented vectors illustrate one value coordinate, not a trained translator.

Inspect the component calculation
Softmax(
  Score vector: Divide(
    Amount: Matrix times vector(
      Rows of coefficients: Keys for the / red / door
      Input vector: Translation step query
    )
    Divisor: Square root(
      Squared value: Key dimension
    )
  )
)
Examples and results
Keys for the / red / door
00
10
01
Translation step query
  1. 02
  2. 10
Key dimension
2
Required result
  1. 00.1636
  2. 10.6728
  3. 20.1636
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

Your last tested pieces can start the full construction. An existing draft there keeps its own pieces.

The query has no preference

This check uses the pieces from “Follow a translation query into its value mixture”.

BOTIE

For the same three source words, Botie's query is now the zero vector. Find the attention output using scaled dot products and softmax. Keep the value entries aligned with the key rows even though the scores tie.
Translation step query
  1. 00
  2. 10
Keys for the / red / door
00
10
01
One value coordinate per source word
  1. 0-3
  2. 16
  3. 212
Key dimension
2

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why can the most attended word change without the output equalling that word's value?

Attention forms a weighted mixture rather than selecting exactly one source row.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the coursePooling a Text →
← Positional Encoding