Stable Diffusion, from first principles

How a neural network learns to turn pure noise into images — and why doing it in a compressed latent space made it run on your GPU.

§1The problem: generating an image is sampling from a distribution

Why care. "Text-to-image" sounds like a UI feature, but underneath it is one of the hardest inference problems in statistics: draw a sample \(x_0\) from the distribution of natural images \(p(x)\), conditioned on a description \(c\). An image is a point in \(\mathbb{R}^{512\times512\times3}\) — 786,432 numbers — and the set of points that look like "a corgi surfing" is a vanishingly thin manifold inside that space. Classic generative families attack this differently: autoregressive models draw one pixel (or patch) at a time — flexible but slow; GANs draw everything in one shot from a latent code — fast but training-unstable (mode collapse, generator/discriminator arms race). Diffusion models take a third route: learn to destroy data with noise, then learn to undo it, one small step at a time.

"Stable Diffusion" specifically = the latent-space diffusion system released open-source in August 2022 by Stability AI, CompVis and Runway, built on the Latent Diffusion Model (LDM) architecture of Rombach et al. (CVPR 2022). The name comes from the company; the design's real claim to fame is that it moved the diffusion process out of pixels into a compressed latent space, cutting the compute by orders of magnitude. Here is the whole system on one map — everything after this section zooms into one box:

• §2 — the forward process: how noise destroys an image, with a closed-form shortcut.
• §3 — the backward process: turning "undo noise" into a simple regression loss.
• §4 — the score view: denoising = following the gradient of log-probability.
• §5–6 — the denoiser network (U-Net) and how text gets injected (cross-attention).
• §7 — the latent-space trick that makes it "Stable" Diffusion.
• §8–9 — sampling fast (DDIM) and steering hard (classifier-free guidance).
• §10 — the family tree: where SD1.5, SDXL and SD3 sit in the larger framework.

§2 The forward process: destroying data with noise

Why start by destroying data? Because you can only learn to reverse a process you can simulate forward cheaply. The forward process is a fixed (untrained) Markov chain that gradually blends the image with Gaussian noise. At step \(t\) it does:

Every symbol: \(x_0\) is the real image, thought of as a vector; each pixel channel is one coordinate. \(\be{t}\in(0,1)\) is a small noise variance schedule — how much variance of new noise to inject at step \(t\). The factor \(\sqrt{1-\be{t}}\) shrinks the old signal slightly so the overall pixel variance stays \(O(1)\) instead of exploding as noise accumulates. \(T=1000\) in Stable Diffusion. After \(T\) steps, \(x_T\) is statistically indistinguishable from pure \(\mathcal{N}(0, I)\) noise.

2.1 The shortcut: any noised version, in one step

Why we need it: during training we will want a randomly chosen \(x_t\) from a random image. Simulating 1..t steps each time would be wasteful — and unnecessary, because the chain has a closed-form marginal. Expand two steps by hand (each step: shrink by \(\sqrt{1-\be{}}\), add fresh noise; independent Gaussians add in variance):

where \(\al{t} := 1-\be{t}\) and \(\bar\Eps \sim \mathcal{N}(0,I)\) is the sum of the two noises, re-standardized (variances \(\al{1}\be{2} + \be{2}\) add to \(1-\al{1}\al{2}\) because \(\Eps_1\) was first shrunk by \(\sqrt{\al{2}}\)). The pattern generalizes — define the cumulative product

So one Gaussian sampling act gives you any timestep directly:

This one line is the workhorse of everything that follows: training samples \(t\) uniformly and applies it; the interactive demo below applies it; and the denoising network's job, given \(x_t\), is precisely to estimate the hidden \(\Eps\).

2.2 Watch it happen — the real SD schedule on a real image

\(t=0\) · \(\abar{t}=1\)

\(t=250\) · \(\abar{t}=0.675\)

\(t=500\) · \(\abar{t}=0.278\)

\(t=750\) · \(\abar{t}=0.057\)

\(t=1000\) · \(\abar{t}=0.0047\)

Figure 2 — Forward diffusion. A synthetic image noised with the exact Stable Diffusion v1.x schedule (\(\be{1}=0.00085 \to \be{T}=0.012\), scaled-linear, \(T=1000\)), using one fixed noise draw \(\Eps\) for all frames — only the weights \(\sqrt{\abar{t}}\) and \(\sqrt{1-\abar{t}}\) change. At \(t=1000\), \(\sqrt{\abar{T}}=0.068\): only 6.8% of the original signal survives.

2.3 The schedule is a design choice

Why the schedule matters: \(\abar{t}\) is the fraction of the original signal's variance that survives at step \(t\). The schedule decides how fast that fraction dies — i.e. how many timesteps are spent in the "mostly signal", "half-and-half", and "mostly noise" regimes. Stable Diffusion v1.x uses a scaled-linear schedule: \(\be{t}\) is the square of a line from \(\sqrt{0.00085}\) to \(\sqrt{0.012}\). Squaring bends the curve so \(\abar{t}\) stays high early and collapses late — more training/sampling steps land in the high-noise regime where global structure is decided.

Figure 3 — Signal survival \(\abar{t}\). Blue: SD v1.x scaled-linear schedule (computed exactly from the released configuration). Orange: the cosine schedule of Improved DDPM, an alternative that keeps more signal late. The SD schedule halves the signal around \(t\approx355\) and leaves only 0.47% of it at \(t=1000\).

Exact values from the SD v1.x schedule (computed by direct multiplication of the 1000 steps):

Later work (Nichol & Dhariwal's Improved DDPM) proposed the cosine schedule \(\abar{t} = f(t)/f(0)\) with \(f(t)=\cos^2\!\big(\tfrac{\pi}{2}\cdot\tfrac{t/T+s}{1+s}\big)\), \(s=0.008\), which dies more evenly and is common in latent models (SDXL's refiner, for example, uses cosine-like schedules).

§3 The reverse process: turning "undo noise" into a regression problem

Why this is the heart of the method: generation is the reverse process — we must start from \(x_T\sim\mathcal{N}(0,I)\) and walk back to an image. Bayes' rule gives the one-step reverse transition:

The first factor is analytic (Gaussian × Gaussian, via Bayes). The second factor, \(q(x_0\mid x_t)\) — "which real images could have produced this noise blob?" — is exactly the intractable distribution over the entire dataset that we were trying to avoid characterizing. So the reverse transition cannot be computed in closed form. The move: approximate it with a Gaussian \(p_\theta(x_{t-1}\mid x_t) = \mathcal{N}(x_{t-1};\, \mu_\theta(x_t,t),\, \sigma^2_t I)\) whose mean \(\mu_\theta\) is a neural network we learn.

3.1 The variational objective, collapsed

Why variational: "make \(p_\theta\)'s reverse chain end up at images" needs a scalar objective. The diffusion model is a latent-variable model with the forward process as a fixed encoder, so the natural objective is the variational lower bound (ELBO) on the data log-likelihood — the same machinery as a VAE, just with \(T\) stochastic layers. Writing it out and using the Markov structure, the bound telescopes into a sum of one-step terms (each KL divergence between two Gaussians whose means differ only through \(\mu_\theta\)):

Each KL is between two Gaussians with the same variance \(\tilde\beta_t\), so it is just a scaled squared distance of means: \(\frac{\tilde\beta_t}{2\sigma^2}\|\mu^\star_t - \mu_\theta(x_t,t)\|^2\), where \(\mu^\star_t\) is the true posterior mean. Training = sample \(x_0\), sample \(t\), run the forward shortcut to get \(x_t\), and regress the network's mean toward the analytic target. That's already trainable — but the target reparametrizes beautifully. The true posterior mean can be written (solve the two-Gaussian Bayes product) as

and since \(x_t = \sqrt{\abar{t}}\,x_0 + \sqrt{1-\abar{t}}\,\Eps\), we can substitute \(x_0 = \frac{x_t - \sqrt{1-\abar{t}}\,\Eps}{\sqrt{\abar{t}}}\). The \(x_t\)-terms then cancel algebraically and the target mean becomes a function of the noise:

So instead of predicting a mean pixel-by-pixel, the network can predict the noise vector \(\Eps\) that was mixed in; the mean follows mechanically. Ho et al. (DDPM, 2020) found that training this parametrization with the KL weights dropped to 1 works better in practice, leaving the beautifully simple objective that Stable Diffusion actually trains with:

Read it as an algorithm, one training step: (1) grab an image–caption pair \((x_0, c)\); (2) roll a timestep \(t\) and a noise cube \(\Eps\); (3) mix them with the closed form; (4) ask the network "what was \(\Eps\)?" — also feeding it \(t\) and the text \(c\); (5) MSE gradient step. Nothing else: no adversary, no value function, no curriculum beyond uniform \(t\).

§4 What the network actually learns: the score, i.e. where probability increases

Why a second viewpoint helps: "predict the noise" is operational but opaque. There is a deeper object hiding inside \(\epsT\): the score \(s(x,t) := \nabla_{x}\log q_t(x)\) — the gradient of the log-density of the noised marginal. It points from any \(x_t\) toward regions of higher probability under \(q_t\). The identity tying noise prediction to the score needs one computation: differentiate \(\log q_t\) using the mixture form \(q_t(x) = \int \mathcal{N}(x;\sqrt{\abar{t}}x_0, (1-\abar{t})I)\, p_{\text{data}}(x_0)\, dx_0\). Differentiating under the integral and recognizing the posterior expectation:

The left side of the implication is Tweedie's formula for a Gaussian channel: the posterior-mean clean image equals the noisy input plus a variance-scaled push up the log-probability gradient. Comparing with the definition \(x_t = \sqrt{\abar{t}} x_0 + \sqrt{1-\abar{t}}\,\Eps\), i.e. \(\hat\Eps = \frac{x_t - \sqrt{\abar{t}}\,\hat x_0}{\sqrt{1-\abar{t}}}\), we get the punchline:

Predicting the noise and estimating the score are the same job, up to a known scale factor. This unlocks the geometric picture: denoising = taking a small step along \(s_\theta\), the empirical "which-way-is-more-image-like" compass. At \(t\) large, \(q_t\) is nearly \(\mathcal{N}(0,I)\): the score is weak and smooth, pointing at coarse data structure — any image-ish blob. At \(t\) small, the score has sharp, local basins around actual images: it only fixes fine detail. The chain of steps is therefore a curriculum: global layout first, texture last. The figure below computes the exact score of a 2-D mixture of three Gaussians at three noise levels — the toy is analytic, but the shape of the field is exactly what a trained \(\epsT\) implements for images.

Figure 5 — The learned vector field, made exact. Arrows show \(\nabla_{x}\log q_\sigma(x)\) for a 3-mode Gaussian mixture convolved with noise of width \(\sigma = 0.15, 0.6, 1.5\) (left→right); gray contours are the corresponding densities; arrow scale is shared across panels. At low noise the field splits into three strong attractors (fine detail); at high noise it is one weak pull toward the data mass as a whole (global layout). A diffusion sampler walks downhill in this field while the field itself anneals.

This also explains the sampler we will derive in §8: the DDPM update is (up to constants) score ascent + fresh noise — annealed Langevin dynamics. And it clarifies what the network must not do: it never produces an image directly; it only produces a direction field, and the iteration does the rest.

§5 The denoiser is a U-Net — same size in, same size out, at every scale

Why a U-Net and not a plain CNN or transformer: the network's contract from §3 is odd by deep-learning standards: it takes a full image-shaped tensor \(x_t\) and must return a full image-shaped noise estimate \(\hat\Eps\), while being able to reason about structure at every scale — from global composition (is this a beach?) to pixel texture (is this foam?). Two architectural facts deliver that. (1) Convolutions make the output shape free: a conv stack with same-padding maps \(H{\times}W\) to \(H{\times}W\) — the input/output contract costs nothing. (2) The encoder–decoder "U" with skip connections gives multi-scale reasoning with clean detail: the down-sampling path compresses spatially into a semantic summary (small spatial size, many channels), the up-sampling path re-expands it, and the horizontal "skip" arrows paste high-resolution features from the encoder directly across — so fine detail never has to survive the bottleneck. Diffusion U-Nets add two ingredients:

• Time embedding. The same weights must answer a different question at every noise level, so \(t\) must be injected everywhere. \(t\) is first mapped by hand to a sinusoidal vector (like the positional encodings of transformers), then a small MLP turns it into a scale-and-shift pair \((\gamma_t, \beta_t)\) that modulates every residual block of the U-Net: \(h \mapsto \gamma_t \odot h + \beta_t\). This is why one network spans the whole curriculum of §4 instead of one network per \(t\).
• Attention blocks. Convolutions mix nearby pixels only. At low resolutions (where each position "sees" a large chunk of the image) the U-Net inserts self-attention so distant regions coordinate — the sun's glow may need to agree with the sea's highlight on the other side of the image. In SD v1.5 these transformer blocks appear at resolutions \(64, 32, 16, 8\) of the latent grid, and — crucially — they are cross-attention, wired to the text (§6).

§6 Text enters through cross-attention: every image patch reads the prompt

Why not just concatenate the prompt: generation must bind which words affect which image regions — "red hat" touches only the hat; "surfing" sets the pose everywhere. A single global vector cannot express that routing. The mechanism that can is cross-attention, already the standard binding operator in transformers. The prompt is first encoded by a frozen CLIP ViT-L/14 text transformer into a sequence of 77 token embeddings, each 768-dimensional: \(c \in \mathbb{R}^{77 \times 768}\) (prompts are padded/truncated to exactly 77 tokens — the reason prompts "run out" around ~75 words). Inside each attention block of the U-Net:

with \(h\) = the U-Net's spatial features (flattened to \(hw\) rows) and \(c\) = the text tokens. The softmax rows are per-image-position probability distributions over prompt tokens: each patch literally asks "which words are relevant to me right now?" and receives the corresponding \(V\)-values mixed accordingly. Through thousands of such layers during training, token↔region bindings ("corgi" ↔ furry blob here) become consistent across the dataset. Note the asymmetry — this is why attention beats concatenation: the same text embedding is re-queried at every position and every layer, so one token can influence many regions and one region can consult many tokens.

§7 The "Stable" in Stable Diffusion: run the whole thing in a compressed latent space

Why pixels are the enemy: take the loss of §3 and the architecture of §5 at their word and train on \(512{\times}512\) RGB images. The self-attention blocks alone are fatal: attention cost scales quadratically with the number of tokens, and a \(512{\times}512\) image has \(262{,}144\) positions, versus \(4{,}096\) for a \(64{\times}64\) grid — \(4096\times\) more attention entries (6.9×10¹⁰ vs 1.7×10⁷), and hundreds of forward/backward passes per training step besides. Rombach et al.'s insight (LDM): most of those pixels are perceptually irrelevant detail. So first train a small autoencoder \((\mathcal{E}, \mathcal{D})\) to compress images into a latent \(z = \mathcal{E}(x)\), then run the entire diffusion machinery of §2–6 on \(z\) instead of \(x\). Decode at the very end with \(\mathcal{D}\).

§8 Sampling: DDPM's 1000 steps, DDIM's 25, and the ODE hiding underneath

Why sampling needs its own section: training gave us \(\epsT\) — a one-step denoiser. Generation must call it repeatedly, and how we chain the calls determines speed, diversity, and even determinism. The ancestral DDPM sampler follows the true reverse chain one step at a time. Using \(\hat x_0(z_t) = \frac{z_t + (1-\abar{t})\, s_\theta(z_t,t)}{\sqrt{\abar{t}}}\) (Tweedie, §4) it is easiest to read as: estimate the clean image, take a noise-blurred partial step toward it:

z ~ N(0, I)                                  # start from pure noise,  t = T
for t = T, T-1, ..., 1:                      # 1000 iterations, 1000 U-Net calls
    ŝ   = score_θ(z_t, t)                    # (or ε̂ from the U-Net: ŝ = -ε̂/√(1-ᾱ_t))
    x̂₀  = (z_t + (1-ᾱ_t)·ŝ) / √ᾱ_t          # best current guess of the clean image
    μ   = (√ᾱ_{t-1}·β_t·x̂₀ + √α_t·(1-ᾱ_{t-1})·z_t) / (1-ᾱ_t)   # posterior mean
    z_{t-1} = μ + √β̃_t · η,  η ~ N(0, I)     # β̃_t = (1-ᾱ_{t-1})/(1-ᾱ_t)·β_t
return z₀                                    # last step has zero added noise

Each step re-injects fresh noise \(\eta\) — that stochasticity is what lets one model produce different images per seed, and it is exactly the annealed-Langevin dynamics predicted by the score view of §4. The figure below runs this exactly: the data is a two-mode 1-D distribution (gray, bottom), the score is computed analytically at every step, and 16 sampled trajectories are drawn from \(t=1000\) (noise, top) to \(t=0\) (data, bottom):

8.1 DDIM: big deterministic steps on the same field

Why 1000 calls is absurd: 25 images/minute on a good GPU. Song et al. (DDIM, 2021) asked which family of reverse processes has the same marginals \(q(x_t)\) but doesn't need step-by-step noise. Answer: a family indexed by a variance parameter \(\sigma^2\); DDPM sits at one end (\(\sigma^2 = \tilde\beta_t\)), and the deterministic end \(\sigma^2 = 0\) gives the DDIM update — which, beautifully, is just "project the current Tweedie estimate \(\hat x_0\) onto the next noise level":

No new noise is added: the trajectory is a function of the seed alone. And nothing forces \(t{-}1 = t{-}\)1: you may jump \(1000 \to 960 \to 900 \to \cdots\) — the model was never told the steps are adjacent. In practice SD ships with 20–50 DDIM steps and the quality loss is small. The deeper reason this works: the deterministic process traces the probability-flow ODE — the noiseless vector field whose flow maps \(\mathcal{N}(0,I)\) onto the data distribution with the same marginals as the SDE (§4's score is its right-hand side up to constants). DDIM is a coarse numerical integrator of that ODE; stochastic DDPM integrates the SDE. Same learned field, two integrators — and the ODE view is what makes lossless img2img possible: encode a real image, run the ODE forward to intermediate \(t_0 < T\), then decode with a new prompt — a deterministic round trip through "noise" that keeps the composition.

§9 Classifier-free guidance: extrapolating toward the prompt

Why guidance at all: a sampler that draws from \(p(x \mid c)\) faithfully draws typical images for the prompt — including the bland majority of the dataset. Users want on-prompt, striking images: a knob that trades diversity for prompt-adherence. The original trick (classifier guidance, Dhariwal & Nichol 2021) trained a separate image classifier on noised images and pushed samples along \(\nabla_x \log p(c \mid x_t)\) — extra model, extra gradients. Ho & Salimans (2022) noticed the classifier is unnecessary: train one conditional model that occasionally ignores its condition, and the classifier's score can be assembled from the difference of two predictions it already makes. Concretely: during SD training, the text embedding \(c\) is replaced by a learned empty-prompt embedding \(\varnothing\) with probability 10%. Then \(\epsT(x_t,t,c)\) and \(\epsT(x_t,t,\varnothing)\) estimate the conditional and unconditional scores, and since (Bayes) \(\nabla \log p(x\mid c) = \nabla\log p(x) + \nabla \log p(c \mid x)\), the difference of the two scores is the classifier's gradient for free. At sample time, overweight it:

This is a linear extrapolation: at \(\gamma=1\) you recover the conditional score; \(\gamma > 1\) pushes beyond it along the direction "what the prompt adds over no prompt". The two U-Net evaluations are batched, so guidance costs ~2× compute per step — the single most important quality knob in the entire system.

§10 The family tree: where SD sits, and what came after

Why zoom out: every piece you now know is one node in a larger, coherent framework — and the lineage explains the models you'll meet next.

10.1 The bigger framework — three equivalent descriptions

Why the same object has three names: it clarifies what is essential. (1) Hierarchical VAE view: diffusion is a VAE with \(T\) fixed Gaussian "encoder" layers (the forward process) and one learned decoder chain — §3's ELBO is literally the VAE bound. (2) Score-based SDE view: as \(T \to \infty\), forward diffusion becomes a stochastic differential equation \(dx = -\tfrac{1}{2}\beta(t)\,x\,dt + \sqrt{\beta(t)}\,dW\); sampling its time-reversal is generation, and the network estimates the reversal's drift (the score) — §4. (3) ODE / flow view: the same marginals are produced by a deterministic ODE (the probability-flow equation), which DDIM integrates and img2img exploits. Modern rectified flow (SD3, Flux) trains the ODE view directly: regress the constant velocity \(v = x_{\text{data}} - \Eps\) along straight interpolation lines from noise to data — straighter paths integrate in fewer steps, which is the same reason DDIM beats DDIM-free samplers on curved VP trajectories.

Appendix — Stable Diffusion v1.5, the concrete configuration

Further reading (the primary sources, in reading order)

1. Ho, Jain, Abbeel — Denoising Diffusion Probabilistic Models (2020): §3's derivation, in full.
2. Song et al. — Score-Based Generative Modeling through SDEs (2021): the unifying SDE/ODE view of §4/§8.
3. Song, Meng, Ermon — DDIM (2021): non-Markovian forward processes and fast sampling.
4. Ho & Salimans — Classifier-Free Diffusion Guidance (2022): §9.
5. Rombach et al. — High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2022): §7 — the SD architecture.
6. Esser et al. — Scaling Rectified Flow Transformers (SD3) (2024): where the lineage went next.
7. Sander Dieleman — "The diffusion sampler" (2024): an excellent second pass on everything above.

All numeric claims computed from the released v1.5 configuration: \(\abar{t}\) by direct multiplication of the 1000 \(\be{t}\) values; compression and attention ratios from \(512{\times}512 \to 64{\times}64{\times}4\); Figure 2 and the interactive demo use the exact schedule; Figure 5 uses the analytic score of a Gaussian mixture; Figure 9 simulates the true reverse chain and the DDIM map with that score. Sources for model facts: SD v1.5 model card, LDM paper, CFG paper.