Transformer Blocks

Attention lets tokens exchange information, but that is only part of a transformer. We will put it together with per-token processing, normalisation, and residual connections, explaining the job of each component.

14 sections · 9 questions · 4 coding challenges

1 / 14

The arrow keys move between sections too.

Could Every Position Consult the Others Directly?

BOTIE

A recurrent network carries information through successive steps. When two words are far apart, their relationship may have to survive many intermediate operations. The sequential processing also limits how much of a training sequence can be computed at once.

Vaswani and colleagues introduced the Transformer in 2017 for sequence transduction, including translation. It made attention the main means of exchanging information across positions, without the recurrent or convolutional layers used by many earlier systems.

A block first lets positions gather information through attention. A feed-forward stage then transforms each position's updated representation. Residual connections and normalisation help these stages work together in a trainable stack. The block is useful because it separates communication between positions from processing within a position. A language generator additionally restricts which positions can be consulted, so training does not reveal the next token's answer.

Sources for this history

Vaswani and colleagues (2017), Attention Is All You Need

How Do We Build a Model Around Attention?

Consider the sentence: The cat lay on the blanket because it was tired. To make sense of it, we need to connect the final words with something mentioned earlier. Reading each word in isolation would leave out that connection.

We already have an operation that lets tokens gather information from other tokens: attention. Now we need to place it inside a larger component that can preserve information, combine what it gathers, and pass the result onward. That component is a transformer block.

Step 1 of 4

Give every token a place to keep information

Each token receives a vector, a row of numbers that the model can update. Position information accompanies that row so the model can distinguish the order of the tokens. These rows are the information being processed, not separate little models.

A schematic view of the rows before a block.
Token	Its representation
cat	Row for cat
was	Row for was
tired	Row for tired

cat
Its representation
Row for cat
was
Its representation
Row for was
tired
Its representation
Row for tired

A schematic view of the rows before a block.

Step 2 of 4

Let a row gather relevant context

Attention compares the current row with permitted rows and combines their value vectors. In a decoder used to predict the next token, a row can read earlier positions and itself. It cannot read later positions, because that would reveal the continuation we want it to predict.

Reading was: Highlighted: cat
Highlighted: was
tired
Contribution: Context gathered by attention

The highlighted rows are available to the middle position.

Step 3 of 4

Keep the original information alongside the contribution

Attention contributes an update. We add it to the row that entered the branch, using a residual connection. This gives later operations access to the earlier representation as well as the newly gathered context. Normalization helps control the scale of the values entering the branch.

1
Shortcut
Original row
2
Attention branch
Context contribution
3
Combined
Updated row

The shortcut and attention contribution meet at an addition.

Step 4 of 4

Process what each row now contains

A feed-forward network applies the same learned transformation to each updated row separately. It combines coordinates within a row; attention was the operation that exchanged information between rows. Another residual connection adds this new contribution before the next block.

1
Across positions
Attention gathers context
2
Within each position
Feed-forward network processes coordinates
3
Next block
Receives the resulting rows

Two different jobs inside one block.

Now assemble the block in the order information travels

The numerical example follows those connections through one small block. Change the last token and check that earlier rows cannot change through access to the future.

Its randomly initialized parameters expose the architecture. Learning useful representations requires training, which the next-token lesson explains.

Part 1. Two Jobs Inside the Block

1. Gather context, then process the result

Attention and the feed-forward network do different work. Attention lets each token row gather information from other positions. The feed-forward network then transforms the coordinates within each row, using the same weights at every position.

Let's follow the row for cat. It can read the start token, the word the, and itself. Attention decides how much information to take from each available position. Keep the shares beside the value vectors they multiply.

Available position     Share from the first head
<start>                0.3988
the                    0.3307
cat                    0.2705

Gathered row ≈ 0.3988 × value(<start>)
             + 0.3307 × value(the)
             + 0.2705 × value(cat)

Attention has mixed information from the available positions into this row. What should we calculate from that mixture? The feed-forward network transforms the row's coordinates before the next block uses them.

A learned projection makes a wider intermediate row. Each entry can combine several coordinates from the input.
In this example, ReLU replaces negative intermediate entries with zero. Which entries survive depends on the row.
Another learned projection combines the surviving entries into a row with the original width.

The input-dependent activation matters. Without it, the two projections could be combined into one affine transformation. With it, changing the input can change which intermediate contributions reach the output.

The same weights serve every position because the network’s question is the same wherever a row came from. Where a token sits is already written into its row by the position step, so the network reads the position out of the row rather than needing a separate set of weights for each place. That is also what lets one block read a longer prefix with the same parameters. The live example reads three tokens, and a fourth token would meet the same expansion and contraction matrices.

Separate heads give attention several independently learned comparison spaces. They are not assigned grammatical roles in advance. A head might learn a useful relationship, but its role must be investigated rather than inferred from its name.

2. Preserve a route for the incoming representation

Each branch produces a contribution that is added to its input. These residual additions preserve a direct route for information and gradients. The branch can learn an adjustment without needing to recreate the entire incoming representation.

The cat row through both additions

Follow the third row of the live tables at the default prompt. After the position step the row for cat reads 0.6632, -0.7264, 0.2649, 1.1782. Attention’s contribution to it, which is the difference between the second and fourth tables, is 0.1897, -0.1005, -0.1018, -0.2512, and the feed-forward contribution in the fifth table is -0.5318, -0.1214, -0.1309, 0.3195. Each addition works coordinate by coordinate, and the second coordinate shows the pattern.

-0.7264 + (-0.1005) = -0.8269
-0.8269 + (-0.1214) = -0.9483

The first sum is the second entry of the row after attention and the second sum is the second entry of the row that leaves the block, exactly as the fourth and sixth tables print them. The inputs shown are rounded to four places, so on another coordinate the last digit of a sum can differ from the table by one. Nothing in the second addition removed what the first had put there, and nothing in either removed the row the position step produced. That surviving row is the route the residual additions preserve.

Layer normalization standardizes coordinates within each token row. Our block normalizes before attention and before the feed-forward network. This is a pre-normalization arrangement; the original Transformer paper placed normalization after its residual additions.

Each addition can change the scale of a row, so a branch reads a standardized copy and meets values on the same scale whatever the earlier additions did. The copy is only what the branch reads. Its contribution is added to the un-normalized row, so the direct route is never rescaled inside the block, and the row is normalized once more only before the vocabulary projection in Part 3.

Part 2. Calculating One Decoder Block

3. Write the connections in their execution order

Let X contain the positioned token rows. LN means layer normalization, applied independently to each row. Attention is causal self-attention: each query can read its own position and earlier positions. The first addition produces H.

H = X + Attention(LN(X))

F = ReLU(LN(H) W₁ + b₁) W₂ + b₂
Y = H + F

The first feed-forward matrix expands the row into a wider intermediate representation. ReLU replaces negative intermediate values with zero. The second matrix returns to the model width so its contribution can be added to H. This implementation uses zero biases.

The SDK's attention projections store output units as matrix rows. The two feed-forward matrices shown here use input coordinates as rows, so they multiply on the right without a transpose. Their shapes, rather than their letters, determine the multiplication.

4. Check the dimensions before adding

The live block uses four coordinates per token, two attention heads, and eight intermediate feed-forward coordinates. For one token row, the transformations have these shapes:

Input row:           (1, 4)
Expansion W₁:        (4, 8)
Intermediate row:   (1, 8)
Contraction W₂:     (8, 4)
Branch output:      (1, 4)
Residual output:    (1, 4)

The final row can enter another block with the same model width. A vocabulary projection after the final block turns these hidden coordinates into next-token scores.

Part 3. From a Block to a Language Model

5. Follow a NumPy forward pass

The reference decoder combines embeddings, sinusoidal positions, one pre-normalized block, final row normalization, and a vocabulary projection. It exposes its intermediate arrays so the causal relationship can be checked.

import numpy as np
from oop_ml.numpy.modern import CausalLanguageModel

model = CausalLanguageModel(n_vocab=7, n_model=4, n_heads=2)
trace = model.respond_to(np.array([0, 1, 2]))
print(trace.attention)
print(trace.logits)

The check the live tables let you make

Set the last token to dog. The earlier rows for <start> and the remain unchanged in this example because the causal mask prevents them from reading the third position. The third row changes. In the first head, its weights move from [0.3988, 0.3307, 0.2705] for cat to [0.1624, 0.4041, 0.4335] for dog. Follow that row through the later tables to see how the changed context representation propagates.

This object implements inference with fixed seeded parameters. It does not implement whole-decoder training, cross-attention, padding masks, or a key/value cache. The attention primitive itself already supports differentiation; composing a complete training pipeline is a further step.

6. Distinguish the architecture from the learned behavior

Stacking blocks creates repeated opportunities to gather and transform information. Training determines which representations and comparisons are useful. Randomly initialized blocks have the structure of the computation but have not learned grammar, facts, or instruction following.

A decoder-only language model is one transformer arrangement. Encoders can use unmasked attention, and encoder-decoder models add cross-attention to another sequence. The causal decoder here is the arrangement we need for the next lessons.

The original paper introduces the attention-based encoder-decoder architecture. Our one-block causal example uses the pre-normalization variant described above. Attention Is All You Need.

Continue with Next-Token Prediction.

A Batch Adds an Axis, Not a Conversation between Examples

A decoder batch commonly stores batch, token and feature axes. Attention should compare positions within the same sequence, not words from unrelated examples in other batch rows.

Different sequence lengths need padding masks, while next-token training also needs a causal mask. Check both with unequal lengths. A triangular mask alone prevents looking forward but does not automatically remove every padded key.

The Feed-Forward Network Reuses Its Rule at Every Position

Attention mixes information between positions. The feed-forward network applies the same learned transformation independently to each position's current vector. Expanding and then reducing the feature width gives that transformation room to form nonlinear combinations.

This distinction explains why the block needs both operations. Sharing the same feed-forward weights across positions does not mean each position produces the same output.

Apply one position-wise transformation to two rows

The supplied matrices isolate the expand, activate and contract sequence.

Python

import numpy as np
X = np.array([[1.,-1.],[2.,1.]])
W1 = np.array([[1.,0.,1.],[0.,1.,-1.]])
W2 = np.array([[1.,0.],[0.,1.],[1.,1.]])
hidden = np.maximum(X@W1, 0)
print(hidden.tolist())
print((hidden@W2).tolist())

Output

[[1.0, 0.0, 2.0], [2.0, 1.0, 1.0]]
[[3.0, 2.0], [3.0, 2.0]]

The rows happen to produce equal final vectors despite different hidden activations. Shared weights do not require equal outputs, but they can discard distinctions.

A Block Still Needs a Vocabulary Head and an Objective

A decoder language model begins with token representations, applies a stack of blocks, and maps each final position to one score per vocabulary entry. The shifted target token supplies the training label at that position.

Gradients must pass from that loss through the vocabulary projection, every block and the input embeddings that participated. A forward pass through one block demonstrates architecture. It is not a pretrained language model and has no learned linguistic competence by itself.

Normalisation Placement Is Part of the Architecture

Applying normalisation before a sublayer and applying it after the residual addition produce different computations and gradient paths. Preserve the selected convention when reproducing a model; moving a normalisation is not a formatting change.

Training and generation also require different execution behaviour for dropout and gradient tracking. Save enough configuration to reconstruct the exact block, including widths, head count, activation, normalisation convention and mask rules.