Diffusion Models

Adding noise to an example is easy to control. Learning to remove it gives us a useful training problem, and a trained reverse process can then turn a noisy starting point into a generated example.

16 sections · 10 questions · 4 coding challenges

1 / 16

The arrow keys move between sections too.

Could Learning to Remove Noise Teach a Model to Generate?

BOTIE

Creating a detailed image in one operation is a difficult task. Adding a little noise to an existing image is easy, and learning to account for a small corruption gives us a more manageable question. What if we could organise generation as many such steps?

Sohl-Dickstein and colleagues' 2015 diffusion work modelled data through a gradual noising process and a learned reverse process. Ho, Jain and Abbeel's 2020 work developed a particularly effective denoising formulation for image generation.

During training, we corrupt examples by known amounts and teach a model to estimate the noise or an equivalent denoising quantity. During generation, we start with noise and repeatedly apply learned reverse steps. Each step is conditioned on where we are in the process. This does not recover a uniquely hidden original image from arbitrary noise; it samples a new example using regularities learned from the training data.

Sources for this history

Can Learning to Remove Noise Teach a Model to Create?

Imagine adding a little random interference to a simple picture, then adding more until its structure is difficult to see. Creating corrupted examples is easy because we control the noise. Could those examples teach a model how to move in the other direction?

Diffusion models use this idea to construct a learning problem. We start with real examples, create noisy versions, and train a model to estimate information needed to remove noise. Generation then begins with random noise and repeatedly uses a learned reverse process.

Step 1 of 4

Begin with a recognizable signal

Take a small picture with a bright region against a dark background. Its pixel values are the clean example we want the training process to preserve information about. We know those values when preparing training data.

Clean signal

Illustrative pixel grids, not outputs of a trained diffusion model.

Step 2 of 4

Mix in noise at a known strength

A schedule specifies how strongly to corrupt the example. Early steps retain more of the clean signal. Later steps retain less. We keep track of the step because a lightly corrupted input and a heavily corrupted input need different treatment.

Clean

Corrupted

A schematic corruption, with the same pixel positions kept visible.

Step 3 of 4

Turn our known noise into a training target

Because we created the corruption, we know which random noise was used. One common training objective asks a network to predict that noise from the noisy input and the step. It is learning a statistical estimate across many examples, not receiving the exact hidden noise at generation time.

Network inputs: Noisy example
Noise step
Network output: Predicted noise
Training target: Actual noise used

The target comes from the corruption process.

Step 4 of 4

Use a learned estimate repeatedly when generating

At generation time there is no original clean picture to consult. Start with random noise and apply reverse updates using the trained predictor. Each update uses the current signal and its noise level. An estimate at one step is part of a sequence, not a guarantee of perfect recovery in one move.

1
Start
Random noise
2
Repeated update
- Predict
- Take a reverse step
3
Finish
Generated sample

The reverse route needs a trained model.

Separate known training noise from a model's estimate

The example supplies a controllable fraction of the exact noise as an oracle. It shows why an accurate estimate matters and why heavy corruption can amplify estimation errors.

It does not train a denoiser or run a reverse sampler. The equations below identify where those components belong.

Part 1. Construct the Supervision Ourselves

1. Define a forward corruption schedule

If we want to generate a new picture, we cannot supply one uniquely correct image for every random starting point. We can, however, take an existing picture and add noise whose value we know. That gives us a noisy input with a training target. Diffusion models learn from these deliberately corrupted examples, then use the learned denoising behaviour to generate from fresh noise.

A forward step reduces the previous signal slightly and adds Gaussian noise. The schedule sets how much noise enters at each step. After enough corruption, the resulting distribution can become close to a simple Gaussian distribution.

Our first example is a strip of six pixels, [0, 0, 1, 1, 0, 0], with two bright pixels between dark ones. We use twenty equal-strength noise steps. The first scales the signal by about 0.9487 and adds a Gaussian draw scaled by about 0.3162. How much an individual pixel changes depends on the draw. Repeating the process gradually reduces the original signal's contribution.

forward step	multiplier on the clean strip	multiplier on the noise
0	1	0
1	0.9487	0.3162
5	0.7684	0.6399
10	0.5905	0.8070
20	0.3487	0.9372

What the strip is made of after a given number of steps. Part 2 derives both columns from the schedule.

The model does not need to learn this forward process because we specify it. Its purpose is to supply noisy inputs with known relationships to clean examples. The small schedule on this page demonstrates the relationship but is not long enough to imply that all signal is gone.

The last row of the table is that caveat in numbers. After all twenty steps the clean strip still enters with a multiplier of 0.3487, so the bright pixels are faint and not gone. The same per-step noise would have to run for 44 steps before the retained fraction of the signal, which is the square of that multiplier, fell below one percent.

2. Train a predictor that knows the noise level

A common denoising network reads the noisy sample and the timestep, then predicts the noise used in its construction. A timestep embedding gives the network information about how much corruption to expect.

One training example on this page looks like this. The clean strip and one noise draw are combined at step 5. The network is shown the noisy strip and the number 5, and the answer it is graded against is the draw.

pixel	clean value	noise draw, the target	noisy value at step 5, the input
1	0	−0.6518	−0.4171
2	0	−0.1747	−0.1118
3	1	1.6637	1.8331
4	1	0.6591	1.1902
5	0	−1.6414	−1.0504
6	0	−0.0052	−0.0033

The live example at its default step. The clean column is used to build the input and is then put away.

Nobody labeled anything. The target column came from a random number generator a moment before the input was built from it. A new draw or a new step makes a new training example from the same clean strip, so one clean example supplies as many pairs as we care to construct.

The step has to be an input because the noisy values do not reveal it. On this draw the third pixel reads 1.8331 at step 5 and 1.9080 at step 20, nearly the same number. At step 5 the clean signal entered with a multiplier of 0.7684 and at step 20 with 0.3487, so those two similar readings contain very different amounts of noise. A predictor that was not told the step could not know which case it was looking at.

A U-Net can serve as this predictor because it combines broad image context with fine spatial detail. Its diffusion target is the known added noise. This differs from the semantic segmentation task, where U-Net is trained with labeled images and corresponding target masks. Conditional diffusion can also use text or class labels.

Part 2. Calculate a Noisy Training Pair

3. Jump directly to a chosen forward step

Let β be the per-step noise variance, α the retained fraction for that step, and ᾱ the cumulative retained fraction. Independent Gaussian additions can be combined into one draw, so we can sample a chosen training step without simulating every preceding step.

αₜ = 1 − βₜ
ᾱₜ = ∏ₛ₌₁ᵗ αₛ

ε ~ Normal(0, I)
xₜ = √ᾱₜ x₀ + √(1 − ᾱₜ) ε

For this example's constant schedule at step 5:
ᾱ₅ = (1 − 0.1)⁵ = 0.59049

The square-root coefficients control the signal and noise scales. The clean sample is x₀, the noisy sample is xₜ, and ε is the known noise draw. At step zero, cumulative retention is one and the signal is unchanged.

Step 5 for two of the six pixels

The cumulative retention at step 5 gives both coefficients. The third pixel is bright and its draw was 1.6637. The first pixel is dark and its draw was −0.6518.

√ᾱ₅ = √0.59049 ≈ 0.7684
√(1 − ᾱ₅) = √0.40951 ≈ 0.6399

Third pixel:  x₅ = (0.7684 × 1) + (0.6399 × 1.6637) ≈ 1.8331
First pixel:  x₅ = (0.7684 × 0) + (0.6399 × (−0.6518)) ≈ −0.4171

Those are the values in the table in step 2, and the other four pixels follow the same line of arithmetic. The two coefficients are tied together. Their squares are ᾱ and 1 − ᾱ, which always total one, so whatever scale the signal gives up the noise takes.

step	cumulative retention ᾱ	signal coefficient √ᾱ	noise coefficient √(1 − ᾱ)
0	1	1	0
1	0.9	0.9487	0.3162
5	0.5905	0.7684	0.6399
10	0.3487	0.5905	0.8070
20	0.1216	0.3487	0.9372

The constant schedule of this page, where every step has a noise variance of 0.1.

The jump is what makes training practical. Building a training example takes one clean sample, one step number and one draw, and the noisy input comes out of a single line of arithmetic, however late the step is.

4. Predict noise and estimate the clean sample

Write the network's noise estimate as ε̂. A common simplified training loss compares the predicted noise with the actual draw. Once an estimate is available, the forward expression can be rearranged to estimate the clean sample.

L_simple = E_{x₀,t,ε}[‖ε − ε̂_θ(xₜ, t)‖²]

x̂₀ = (xₜ − √(1 − ᾱₜ) ε̂) / √ᾱₜ

If the estimate equals the exact noise, this recovers the clean sample. During generation that exact noise is unavailable. A predictor must infer useful information from the noisy input and what it learned from training data.

As signal retention becomes small, division by its square root amplifies noise-estimation errors in this clean-sample estimate. The live oracle control makes that dependence visible.

The estimate with all, half and none of the noise

The live control supplies a chosen fraction of the true draw as the noise estimate. Follow the third pixel at step 5, where the noisy value is 1.8331, the true draw is 1.6637 and the clean value is 1.

All of the draw:   x̂₀ = (1.8331 − (0.6399 × 1.6637)) / 0.7684 ≈ 1.0000
Half of the draw:  x̂₀ = (1.8331 − (0.6399 × 0.8319)) / 0.7684 ≈ 1.6928
None of it:        x̂₀ = (1.8331 − 0) / 0.7684 ≈ 2.3855

The inputs shown are rounded and the results were calculated from the unrounded values. With the whole draw the clean value comes back. With half of it the estimate is too high by 0.6928, and with no estimate at all it is too high by 1.3855. The size of the miss depends on the step as well as on the estimate, which the next lines make exact.

x̂₀ − x₀ = √((1 − ᾱₜ) / ᾱₜ) × (ε − ε̂)

An error in the noise estimate is multiplied by that square root on its way into the clean estimate. Hold the estimate at half of the true draw and move the step.

step	cumulative retention ᾱ	error multiplier	third pixel’s clean estimate
1	0.9	0.3333	1.2773
5	0.5905	0.8328	1.6928
10	0.3487	1.3667	2.1369
20	0.1216	2.6880	3.2360

The clean value is 1 in every row, and the noise estimate is wrong by the same amount in every row, half of the draw.

The same mistake about the noise costs 0.2773 at step 1 and 2.2360 at step 20. Early in the schedule the multiplier is below one and an imperfect predictor still gives a fair picture of the clean sample. Late in the schedule the multiplier is well above one, and a single leap from a very noisy sample to a clean one magnifies whatever the predictor got wrong.

Part 3. What the Reverse Process Adds

5. Distinguish clean estimation from a reverse sampling step

A full diffusion sampler starts from noise and repeatedly predicts a less corrupted state. In the DDPM formulation, one reverse update has the following form. The reverse variance σ² depends on the chosen schedule or model; ξ is fresh standard Gaussian noise and is set to zero on the last step.

μ_θ(xₜ,t) = (1 / √αₜ)
             [xₜ − (βₜ / √(1 − ᾱₜ)) ε̂_θ(xₜ,t)]

xₜ₋₁ = μ_θ(xₜ,t) + σₜ ξ

This is a sequence of learned reverse transitions, not repeated use of the exact forward noise. Different samplers change the update rule and number of steps. The page's numerical object implements forward noising and clean estimation only.

One reverse update beside one clean estimate

Both formulas take a noise estimate, and they do different things with it. Take the third pixel at step 5 again, where the noisy value is 1.8331, and give both formulas the exact draw of 1.6637, so that neither is held back by a poor prediction. On this schedule β₅ is 0.1 and α₅ is 0.9.

Clean estimate:  x̂₀ = (1.8331 − (0.6399 × 1.6637)) / 0.7684 ≈ 1.0000

Reverse mean:    μ = (1 / √0.9) × [1.8331 − ((0.1 / 0.6399) × 1.6637)]
                   ≈ 1.0541 × [1.8331 − 0.2600]
                   ≈ 1.6582

The clean estimate goes all the way to 1 in one move. The reverse update takes the pixel from 1.8331 to about 1.6582, a short distance in the right direction, and then adds fresh noise scaled by σ unless it is the last step. In the formulation above, a sampler repeats that update once for each step of the schedule, which would be twenty times here. It asks the predictor for a new estimate every time, because after each move the sample is a different noisy input at a different step.

Step 4 showed the cost of the alternative. A single leap from step 20 to a clean estimate multiplies the predictor’s error by more than two and a half. The reverse mean above was worked by hand from the formula, with the exact draw standing in for a prediction. Nothing on this page runs a reverse sampler.

6. Reproduce the oracle check

Passing the actual added noise into the recovery method is a useful correctness check. It is not a generative shortcut, because it relies on information that we know only while constructing the training example.

import numpy as np
from oop_ml.numpy.modern import GaussianDiffusion

process = GaussianDiffusion(np.full(20, 0.1))
clean = np.array([0.0, 0.0, 1.0, 1.0, 0.0, 0.0])
noise = np.random.default_rng(4).normal(size=6)
noisy = process.add_noise(clean, noise, step=5)
recovered = process.recover_clean(noisy, noise, step=5)
print(recovered)

Run as written, the code returns the clean strip, [0, 0, 1, 1, 0, 0], to within rounding in the sixteenth decimal place. Replace the true draw with zeros, which is what a predictor that had learned nothing might offer, and the same call returns [−0.5428, −0.1455, 2.3855, 1.5489, −1.3669, −0.0043]. Every pixel is wrong, and the pixels whose draws were largest are wrong by the most.

The check passes only because the script still holds the draw it added a line earlier. When generating there is no clean strip, so no draw was ever added to one, and there is nothing to pass in. The only possible source of a noise estimate is a trained predictor.

To build a working generator around these operations, we would need a trained timestep-conditioned predictor and a reverse sampler. Image quality then depends on that learned model, the training distribution, conditioning, and the sampling procedure.

The paper develops the noise-prediction training formulation and reverse sampling process used in this lesson. Denoising Diffusion Probabilistic Models.

Continue with Evaluating Generative Models.

The Denoiser Must Predict Noise It Was Not Given

The earlier oracle check recovers a clean strip by supplying its actual noise. A learned denoiser receives only the noisy value and the timestep. Its target is the noise used to create that training pair, which stays on the training side of the loss.

To make a complete fit manageable, we will use one-dimensional clean samples instead of images. They come from a Gaussian centred at two with standard deviation one half. A separate linear predictor at each timestep can approximate the optimal noise prediction for this Gaussian teaching distribution.

Give Each Noise Level Its Own Training Pairs

At one timestep, draw clean samples and independent standard-normal noise, combine them using the forward schedule, and fit noise from the noisy value plus an intercept. Repeat for every timestep. The fitted coefficient table represents timestep conditioning in this small example.

An image model usually shares a much richer neural predictor across noise levels and receives a timestep representation. Our separate regressions expose the same input-target separation without pretending to train an image generator.

A Reverse Step Needs a Mean and a Noise Scale

The predicted noise enters the reverse mean. Sampling then adds the chosen reverse noise term, except at the final step. A clean-sample estimate and a reverse sample are different quantities; jumping straight to the clean estimate skips the intended chain.

The program below uses the common DDPM posterior-variance schedule. Starting from a standard Gaussian is appropriate only when the final forward distribution is close enough to it. Sixty steps at the supplied rate make retained signal small for this toy, but finite-step and fitted-model error remain.

αₜ = 1 - βₜ
μθ(xₜ,t) = (xₜ - βₜ εθ(xₜ,t) / √(1-ᾱₜ)) / √αₜ
β̃ₜ = βₜ (1-ᾱₜ₋₁) / (1-ᾱₜ)
xₜ₋₁ = μθ(xₜ,t) + √β̃ₜ z
At the final step, use no additional noise.

Fit the Predictors, Then Generate without a Clean Input

The first loop constructs supervised pairs and fits the timestep-specific predictors. The second loop begins from unrelated Gaussian noise and walks backward using only those fitted coefficients. It never reads a clean target or the original forward noise during generation.

Inspect generated mean and spread against the source distribution. This is a complete toy training and sampling procedure, with a deliberately simple Gaussian source and linear denoisers. It cannot demonstrate image structure, multimodal coverage or text conditioning.

A trained scalar diffusion chain

A fixed generator makes this example repeatable in the browser. Print the sample shape, then its mean and standard deviation rounded to three places. The source mean is two and its standard deviation is one half.

Python

import numpy as np
rng = np.random.default_rng(12)
beta = np.full(60, .1)
alpha = 1-beta
retained = np.cumprod(alpha)
model = []
for t in range(len(beta)):
    clean = rng.normal(2., .5, 20000)
    noise = rng.normal(size=len(clean))
    noisy = np.sqrt(retained[t])*clean + np.sqrt(1-retained[t])*noise
    design = np.column_stack([noisy, np.ones(len(noisy))])
    model.append(np.linalg.lstsq(design, noise, rcond=None)[0])
generated = rng.normal(size=5000)
for t in range(len(beta)-1, -1, -1):
    predicted_noise = model[t][0]*generated + model[t][1]
    mean = (generated-beta[t]*predicted_noise/np.sqrt(1-retained[t]))/np.sqrt(alpha[t])
    previous_retained = retained[t-1] if t > 0 else 1.
    variance = beta[t]*(1-previous_retained)/(1-retained[t])
    generated = mean + np.sqrt(variance)*rng.normal(size=len(generated)) if t > 0 else mean
print(generated.shape)
print(round(float(generated.mean()), 3), round(float(generated.std()), 3))

Output

(5000,)
2.007 0.377

The mean is close to the source mean, but the generated spread is noticeably smaller than one half. The coarse schedule, chosen reverse variance and fitted predictors approximate the reverse process. This is a working chain whose distribution still needs improvement, not a successful fit merely because it produces numbers. Try a finer schedule and compare repeated runs, including spread as well as location.

Check yourself

Questions on Training and Reverse Sampling

Question 1 of 1

Which quantity must generation estimate rather than receive from the training pair?

Use this setup.

Training creates noisy values from clean samples and recorded random noise. Generation begins from a new Gaussian draw and uses the fitted timestep predictors.

Why More Steps and Better Training Solve Different Problems

A finer reverse schedule can reduce discretisation effects, while a more capable denoiser can reduce prediction error. Increasing sampling steps does not automatically fix a predictor trained on the wrong data, and a perfect training fit does not guarantee coverage of unseen modes.

Conditional generation adds another input, such as a class or text representation, and must be trained to use it. Guidance changes the sampling procedure and its tradeoffs. Keep those extensions separate from the unconditional scalar chain so the source of each improvement can be investigated.