Watch the Short: youtube.com/shorts/yYxpjlvW69s

Why the numbers drift

A deep network is a chain of layers. Each layer takes a weighted sum of the activations from the layer before it, passes that sum through an activation function, and hands the result to the next layer.

Training changes the weights at every step. So every layer keeps seeing inputs that have shifted since the last step: a new average, a new spread. In a deep network, those shifts pile up. With a big learning rate, the numbers can drift so far that they blow up, and training dies.

What batch norm does

Batch normalization adds a reset after each layer. For each neuron, over the mini-batch, it does three things:

1. It finds the average (μ_B) and the spread (σ_B) of the neuron's weighted sums.
2. It slides them so their average sits at zero, and squeezes them so their spread is one.
3. It applies a learned scale (γ) and a learned shift (β), so each layer can still pick its own range.

In one line:

y = γ · (x − μ_B) / √(σ_B² + ε) + β

It sits after the weighted sum and before the activation function: Linear → BatchNorm → ReLU. Because β already gives that layer a shift, the layer needs no bias. In conv nets, there is one average and one spread per channel, taken over the mini-batch and every position.

Training and testing

During training, batch norm uses each mini-batch's own average and spread, and keeps running averages of them. At test time it uses those saved running averages. In PyTorch, model.train() and model.eval() switch between the two. The defaults are eps = 1e-5 and momentum = 0.1.

What it bought

Sergey Ioffe and Christian Szegedy, at Google, tested it on Inception, an image network:

• Inception: learning rate 0.0015. 31.0M steps to its best accuracy, 72.2%.
• BN-x5: learning rate 0.0075 (5×). 2.1M steps to reach 72.2%, about 14 times fewer. Best 73.0%.
• BN-x30: learning rate 0.045 (30×). 2.7M steps to reach 72.2%. Best 74.8%.

BN-x5 also used changes that batch norm made possible: no dropout, weight decay cut 5×, a learning rate lowered 6× faster, no local response normalization, better shuffling and fewer photo distortions.

With sigmoid as the activation function, plain Inception never beat 0.1% accuracy, while BN-x5-Sigmoid reached 69.8%.

Our toy test

We tried it on scikit-learn's 8×8 digits, with 1,500 training images and 297 test images. The network had 8 hidden layers of 64 ReLU neurons with a He start, trained by plain SGD on mini-batches of 60.

First, a sweep: three seeds, 1,000 steps each.

• Plain: it trained at learning rates 0.03 and 0.1 (96–99% test accuracy). At 0.3, two of three seeds blew up. At 0.5, all three did. At 1.0, two blew up and the third stayed at 7%. At 2.0, all three blew up.
• Batch norm: every learning rate up to 2.0 trained (96–99%).

Then a race at learning rate 0.5, five times the plain net's biggest safe one, same start, same mini-batches.

• Plain: the training loss fell from 3.2 to 0.67 by step 100, then jumped to 25 at step 130 and hit NaN at step 135. Test accuracy went from 83% at step 100 to 8% at step 132. Other seeds blew up at steps 128 and 348: the exact step is chaotic.
• Batch norm: in layers 2, 4, 6 and 8, the middle stayed at 0.06 or less and the spread stayed between 0.99 and 1.06. Training loss fell to 0.06 by step 125. Test accuracy reached 96% by step 100 and 98% by step 135.

Why does it work?

The paper's idea is "internal covariate shift": each layer's inputs keep changing during training, and batch norm reduces that. Santurkar et al. (2018) tested this and argue that batch norm mostly makes the loss landscape smoother. Bjorck et al. (2018) find that it mainly allows larger learning rates. This is still debated.

Limits and cousins

Small mini-batches hurt it, because their averages are noisy. At a batch size of 2, Group Norm had 10.6% lower error than batch norm (ResNet-50, ImageNet). Transformers use Layer Norm, which normalizes one example at a time.

Try it in PyTorch

nn.Sequential(nn.Linear(256, 256, bias=False), nn.BatchNorm1d(256), nn.ReLU())
nn.Sequential(nn.Conv2d(64, 64, 3, padding=1, bias=False), nn.BatchNorm2d(64), nn.ReLU())

References

• Ioffe, S. & Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ICML 2015, PMLR 37:448–456. https://proceedings.mlr.press/v37/ioffe15.html · https://arxiv.org/abs/1502.03167
• Santurkar, S., Tsipras, D., Ilyas, A. & Madry, A. (2018). How Does Batch Normalization Help Optimization? NeurIPS 2018. https://papers.nips.cc/paper/2018/hash/905056c1ac1dad141560467e0a99e1cf-Abstract.html · https://arxiv.org/abs/1805.11604
• Bjorck, J., Gomes, C., Selman, B. & Weinberger, K. Q. (2018). Understanding Batch Normalization. https://arxiv.org/abs/1806.02375
• Wu, Y. & He, K. (2018). Group Normalization. https://arxiv.org/abs/1803.08494
• Ba, J. L., Kiros, J. R. & Hinton, G. E. (2016). Layer Normalization. https://arxiv.org/abs/1607.06450
• PyTorch docs: BatchNorm1d · BatchNorm2d
• CS878 Deep Learning (Fall 2023), Lecture 12–13: Regularization & Transfer Learning, Batch Normalization slides.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

When a neural network makes a mistake, training works out how much each dial (each weight) was to blame. That blame starts at the answer and travels backward, layer by layer, so every dial knows which way to turn. This is backpropagation.

On the way back, the blame passes through each layer's bend, the activation function. At every layer it gets multiplied by that bend's slope. This is the chain rule: one slope per layer, all multiplied together. In a deep network, that long chain of multiplications can shrink to almost nothing, or grow out of control.

A quarter at a time

Older networks used an S-shaped bend, the sigmoid: σ(s) = 1 / (1 + e⁻ˢ). Its slope is σ(s)(1 − σ(s)). That slope is steepest in the middle, at s = 0, where it is exactly 0.25. Everywhere else it is smaller, and far from the middle it is almost zero (the lecture calls these neurons saturated: they "kill the gradient flow").

So each sigmoid layer passes back at most a quarter of the blame. Ten layers in a row, even at their steepest, pass back 0.25¹⁰ = 0.00000095 of it. That is less than a millionth.

We measured it in a real network: 10 layers of 256 sigmoid neurons with a good (Xavier) start, 512 random inputs and targets, and a squared-error mistake score (numpy's default_rng(0)). Here is the blame that reached each layer's dials, compared with layer 10:

• Layer 10: 1
• Layer 9: 0.19
• Layer 8: 0.051
• Layer 7: 0.013
• Layer 6: 0.003
• Layer 5: 0.00069
• Layer 4: 0.00015
• Layer 3: 0.000035
• Layer 2: 0.0000079
• Layer 1: 0.00000031

The blame shrinks plane by plane: layer 1 gets 0.0000003 of what layer 10 gets.

The first layer gets about three ten-millionths of the blame, so its dials barely move. This is the vanishing gradient problem. Sepp Hochreiter worked it out in his 1991 diploma thesis at the Technical University of Munich, long before deep learning took off. Bengio, Simard and Frasconi analysed the same problem in 1994.

Sepp Hochreiter's 1991 diploma thesis, "Untersuchungen zu dynamischen neuronalen Netzen" (Studies of dynamic neural networks).

Too big: exploding blame

The chain can also go the other way. If the dials start too big, the blame grows at every layer instead of shrinking. We took a healthy ReLU network and made every dial 2× too big. The blame reaching layer 1 was about 500,000× bigger than in the healthy network (512,003× when measured). With dials just 1.5× too big, it was still about 2,000× bigger.

Huge blame means huge steps. The lecture warns that one giant step can throw the dials into a terrible spot with a large mistake, and in the worst case the numbers overflow to Inf or NaN, and training has to restart from an earlier checkpoint. The lecture also points out that this is not only a problem for recurrent networks: it hits all very deep networks.

Gradient clipping: a cap line, and every layer's blame scaled down to fit under it.

The fixes

• ReLU. ReLU is max(0, s). When a neuron is on, its slope is exactly 1, so it passes the blame back at full strength. In the same 10-layer network with ReLU and a He start, the blame stayed between 0.4 and 2.1 of layer 10's at every layer (layer 1: 0.42). It helps a lot, but it is not a cure-all: a neuron that is off passes back nothing, and the dials can still start too big.
• Good starting dials. Starting the dials at the right size keeps the blame level from layer to layer (Deep Learning, Visually #18).
• Batch normalization. It keeps each layer's values in a healthy range during training (coming in #20).
• Skip connections. Shortcuts that let the blame jump over layers, used in ResNets (coming in #35).
• Gradient clipping, for exploding blame. If the total size of the blame is bigger than a limit, scale all of it down to the limit. The direction stays the same; only the step gets smaller.

ReLU: the blame reaches the first layers at full strength.

Try it in PyTorch

Use ReLU layers with a He start, and clip the blame before every step:

import torch
import torch.nn as nn
 
layer = nn.Linear(256, 256)
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")
model = nn.Sequential(layer, nn.ReLU())
 
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

Or watch the blame vanish yourself. This sends one number through ten bends and asks how much blame comes back:

import torch as t
for f in(t.sigmoid,t.relu):
 x=t.ones(1).requires_grad_()
 y=x
 for _ in'x'*10:y=f(y)
 y.backward();print(x.grad)

It prints tensor([2.7848e-07]) for the sigmoid and tensor([1.]) for ReLU. Change 10 to 30: how small does the sigmoid's blame get now?

References

• Hochreiter, J. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Technische Universität München. https://www.bioinf.jku.at/publications/older/3804.pdf
• Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. https://doi.org/10.1109/72.279181
• Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep Sparse Rectifier Neural Networks. AISTATS, PMLR 15, 315–323. https://proceedings.mlr.press/v15/glorot11a.html
• Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML, PMLR 28(3), 1310–1318. https://proceedings.mlr.press/v28/pascanu13.html
• CS878 Deep Learning, SEECS NUST: lecture 10–11 (slides 35–43) and lecture 23–24 (vanishing and exploding gradients, gradient clipping).

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

A neural network learns by turning its dials, the weights. Every dial needs a first value. Why not give them all the same one?

Take a small network: 2 inputs, 4 hidden neurons, 1 output. Set every dial to 0.5 and every bias to 0. Now all four hidden neurons see exactly the same thing, so they compute the same value. They make the same mistake, so training gives each one the same push. After the push they are still identical, so the next step repeats it.

We trained this network for 2,000 steps. The four hidden neurons stayed exactly identical the whole time: the difference between them was 0.0 at every step. They are twins, forever.

Every dial starts at 0.5: the four hidden neurons get the same push at every step. Step 2,000, still twins.

Starting every dial at zero is even worse. Then nothing moves at all: every push is exactly zero, the network answers 0.5 for every input, and the loss stays at 0.693 forever (2 of 4 answers right, no better than a coin).

The corner puzzle (this is XOR)

The puzzle has four corners. The output should be 1 when exactly one input is 1: (0,1) and (1,0) give 1, while (0,0) and (1,1) give 0. This is the XOR problem.

Four twin neurons are no better than one neuron, and one neuron can only draw one straight line. As we saw in part 3, one straight line can't solve XOR. The same-start network gets stuck at 3 of 4: its outputs are 0.00, 0.67, 0.67 and 0.67, so the corner (1,1) is wrong, and the loss is stuck near 0.48.

Four twins act like one neuron: one straight split. 3 of 4 right, the top-right corner is wrong.

Now start the dials at random instead (numpy's default_rng(1)). The first-layer dials are [[0.35, 0.82, 0.33, −1.30], [0.91, 0.45, −0.54, 0.58]], and the output dials come from the same generator. Every neuron begins a little different, so each gets its own push and learns its own part. The network gets all 4 corners right from step 18. By step 2,000 its outputs are 0.00, 1.00, 1.00 and 0.00. Its answer is no longer one straight split: it marks the two corners that should be 1 and leaves a clear diagonal band through the other two.

Random start: four different neurons, four different pushes. 4 of 4 from step 18.

Over 20 random starts (seeds 0 to 19), 19 solve the puzzle.

How random?

Random isn't enough: the size of the random dials matters too. The CS878 lecture tests this on a deep network: 10 layers of 500 tanh neurons each, fed 1,000 random inputs. Every neuron adds up 500 inputs, each multiplied by its dial.

• Dials too small (0.01): the spread of the values shrinks in every layer: 0.21, 0.048, 0.011, 0.0024 and on down to 0.0000003 at layer 10. The signal fades to nothing, and tiny values make tiny pushes, so training crawls.
• Dials too big (1.0): the sums get so large that tanh is pushed to its limit. In every layer, 93% of the values are stuck beyond ±0.95. There, tanh is almost flat, so the pushes vanish again.
• The right size (1/√500 ≈ 0.045): the spread stays healthy all the way down: 0.63, 0.49, 0.41, 0.36, 0.32, 0.29, 0.27, 0.25, 0.24, 0.23. About 7% of the values are stuck in layer 1, 0.4% in layer 2, and almost none after that.

The right dial size: healthy spreads from layer 1 to layer 10.

The rule

A neuron adds up its inputs: s = w₁x₁ + w₂x₂ + … + wₙxₙ. If the dials and the inputs are random with an average of zero, the spreads (variances) add up:

Var(s) = n · Var(w) · Var(x)

To keep the spread the same from one layer to the next, make n · Var(w) = 1, which gives Var(w) = 1/n: dials with a spread of 1/√n. More inputs, smaller starting dials. This is Xavier initialization, named after Xavier Glorot, who worked it out with Yoshua Bengio in 2010. Their paper also balances the backward pass, with Var(w) = 2 / (inputs + outputs). It suits tanh.

ReLU zeroes out half of the values, so ReLU layers need twice as much: Var(w) = 2/n. This is He initialization (Kaiming He and colleagues, 2015).

Try it in PyTorch

import torch.nn as nn
layer = nn.Linear(500, 500)
nn.init.xavier_normal_(layer.weight)                         # tanh layers
nn.init.kaiming_normal_(layer.weight, nonlinearity="relu")   # ReLU layers

For a 500-to-500 layer, xavier_normal_ draws dials with a spread of √(2 / (500 + 500)) = 1/√500 ≈ 0.045, the right size from the Short. kaiming_normal_ uses √(2/500) ≈ 0.063. PyTorch already starts every nn.Linear at random, in a range that shrinks as its inputs grow.

The Short's experiment fits in six lines:

import torch as t;n=500
for s in(.01,1,n**-.5):
  h=t.randn(1000,n)
  for _ in'x'*10:
   h=(h@t.randn(n,n)*s).tanh()
  print(f'{h.std():.1e}')

It prints the spread after 10 layers for each dial size: 3.0e-07 (fades to nothing), 9.8e-01 (stuck at ±1) and 2.3e-01 (healthy). Swap .tanh() for .relu(): which dial size keeps the signal alive now?

References

• Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS, PMLR 9, 249–256. https://proceedings.mlr.press/v9/glorot10a.html
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. ICCV 2015. https://arxiv.org/abs/1502.01852
• CS878 Deep Learning, SEECS NUST, lecture 10–11 (slides 52–72).

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Training an AI is like rolling a ball down a bowl. The ball's position is the AI's dials (its weights), the height is how many mistakes it makes, and the bottom means the fewest mistakes. Each training step moves the ball a little downhill:

w ← w − step size × push

The step size is the learning rate and the push is the gradient. But each push comes from just a few examples (a mini-batch), not all the data, so it always points a little off the true downhill direction.

The bowl in the Short: mistake = ½ · (w1² + 4 · w2²), a long oval with its bottom at (0, 0). The ball starts at (−4, 1.5), 4.27 from the bottom. Each push is the true slope (w1, 4 · w2) plus 0.6 × a normal random number in each direction, from numpy's default_rng(2252). All three runs below use the same noise and take 60 steps.

Big steps bounce, small steps crawl

With a big step size, 0.45, the ball is about 0.2 from the bottom by step 6. Then the noise takes over: near the bottom the true slope is tiny, so the push is mostly noise, and a big step turns it into a big jump past the middle. The ball bounces around and never settles. Over the last 10 steps it is 0.50 from the bottom on average.

Big steps: near the bottom by step 6, then every off push throws the ball past the middle. Step 60, still bouncing.

With a small step size, 0.02, the bounces are tiny, but the trip is slow: 3.33 from the bottom after 10 steps, 2.73 after 20, 1.75 after 40 and 1.21 after 60. Same number of steps, still far away.

Small steps: tiny bounces, but after the same 60 steps the ball is still 1.2 away. The big-step cloud stays as a grey ghost.

Lecture 09 puts it in one line: you need a high learning rate when far away and a low learning rate when close.

Start big, then cut the step size in half

A learning-rate schedule uses both. Start with the big step size, 0.45, and halve it every 10 steps: 0.45, 0.225, 0.113, 0.056, 0.028, 0.014.

The first 10 steps match the big-step run exactly. After that, each halving shrinks the bounces: the ball is 0.25 from the bottom at step 20, 0.14 at step 40 and 0.07 at step 60. Over the last 10 steps it averages 0.09, more than five times closer than big steps alone.

Over 2,000 different noise sequences, the last-10-step averages are: big steps 0.49, small steps 1.31, the schedule 0.09. That is about 5.6 times closer than big steps.

The staircase: 0.45, halved at steps 10, 20, 30, 40 and 50. Big steps end 0.5 away, the schedule 0.09 away.

Other schedules

Halving every n steps is called step decay. Lecture 09 lists other ways to shrink the learning rate α:

• Step decay: multiply α by a decay rate (often 0.5) every n steps.
• Exponential decay: α = tᵉᵖᵒᶜʰ · α0, where the decay rate t is below 1, so α shrinks a little every epoch.
• α = α0 / (1 + decay_rate · epoch): with α0 = 0.1 and decay_rate = 1.0, the learning rate goes 0.1, 0.05, 0.033, 0.025 over epochs 0 to 3.

A popular smooth option is the cosine schedule, which lowers the learning rate along half a cosine wave. SGDR (Loshchilov and Hutter) adds warm restarts: the learning rate decays, jumps back up, and decays again.

Why it works

In 1951, Herbert Robbins and Sutton Monro studied how to find an answer when every measurement is noisy, just like a ball pushed by a few examples at a time. They showed that noisy steps still settle if the step sizes aₜ shrink so that:

• Σ aₜ = ∞: added up, the steps are unlimited, so the ball can still travel as far as it needs.
• Σ aₜ² < ∞: the steps shrink fast enough for the noise they carry to average out.

For example, aₜ = a0 / (1 + k · t) meets both. Halving every few steps is the practical version used in training. Strictly, halving forever adds up to a finite total and misses the first condition, but over a fixed training run it does the same job: big steps far away, small steps up close.

Robbins and Monro, "A Stochastic Approximation Method" (1951): noisy steps settle when the step sizes shrink.

Try it in PyTorch

In PyTorch a schedule wraps the optimizer. This one halves the learning rate every 10 epochs:

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)

Call scheduler.step() once per epoch, after that epoch's optimizer steps. This article's schedule on its own:

import torch as t
o=t.optim.SGD([t.ones(1)],lr=.45)
s=t.optim.lr_scheduler.StepLR(o,10,.5)
for e in range(60):
  if e%10==0:print(e,s.get_last_lr())
  o.step();s.step()

It prints the staircase: 0 [0.45], 10 [0.225], 20 [0.1125], 30 [0.05625], 40 [0.028125], 50 [0.0140625]. Change 10 to 5 and run it again: how small is the step size by step 50 now?

References

• Robbins, H., & Monro, S. (1951). A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3), 400–407. https://doi.org/10.1214/aoms/1177729586
• Loshchilov, I., & Hutter, F. (2016). SGDR: Stochastic Gradient Descent with Warm Restarts. https://arxiv.org/abs/1608.03983
• CS231n notes, "Annealing the learning rate": https://cs231n.github.io/neural-networks-3/
• CS878 Deep Learning, SEECS NUST, lecture 09 (slides 5–7).

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

An AI learns by turning dials. A network has millions of them, its weights, and training nudges every one a little at a time. To know which way to turn a dial, training measures how the mistake changes when that dial moves. That measurement is the dial's push, the gradient. It points downhill, toward a smaller mistake.

Plain gradient descent moves every dial with the same rule:

w ← w − step × push

There is one step size, and every dial shares it. The trouble is that hills differ. Take two dials. Both start at w = −2, and both make the smallest mistake at w = 0.

• The top dial sits on a steep, narrow hill: mistake = 25 · w². Its push at the start is 100.
• The bottom dial sits on a wide, nearly flat hill: mistake = ½ · w². Its push is 2.

Same position, but on the steep hill the push is fifty times bigger.

Two dials, two hills: the same start, but the steep hill's push is fifty times bigger, and one slider sets the step size for both.

Try a big step size, 0.045. The steep dial jumps right past the bottom: from −2 to 2.5, then −3.125, then 3.906. Every jump is bigger than the last, and it flies off. Try a small step size, 0.015, which suits the steep hill: it settles in three hops (−0.5, −0.125, −0.031). Now the flat dial crawls. It needs 245 steps to get within 0.05 of the bottom. No single step size fits both.

With a big step size the steep dial jumps past the bottom, higher each time, and flies off.

Each dial's own step size

The fix is to stop sharing. Each dial remembers how big its pushes usually are, and divides its push by that. A dial with big pushes takes careful steps. A dial with tiny pushes takes bolder ones.

On the very first step, each push divided by its usual size is exactly 1 for both dials (100 out of 100, and 2 out of 2). So both dials move exactly the same distance: 0.1. After that their paths stay identical. They roll down together, overshoot a little to +0.26, swing back, and both stay within 0.05 of the bottom from step 71.

Both dials take the same steps: a line joining them stays vertical as they roll down together.

What Adam computes

This method is Adam, from a paper by Diederik Kingma and Jimmy Ba, published at ICLR 2015. The name comes from "adaptive moment estimation". It combines two ideas: momentum, from the last episode, and its own step size for every dial.

For each dial, g is its push at this step and t counts the steps (1, 2, 3, …). Adam keeps two running averages and turns them into a step:

• m = 0.9 · m + 0.1 · g (the average push: this is momentum)
• v = 0.999 · v + 0.001 · g² (the average squared push: how big the pushes usually are)
• m̂ = m / (1 − 0.9ᵗ) and v̂ = v / (1 − 0.999ᵗ) (m and v start at zero, so early on they come out too small; this corrects them)
• step = lr · m̂ / (√v̂ + 1e-8)

The 1e-8 is only there to avoid dividing by zero. The key part is the division by √v̂, which has the same size as the push. If a dial's pushes are all fifty times bigger, its m̂ and its √v̂ are both fifty times bigger, and they cancel. That is why the steep dial and the flat dial move the same.

The steeper-or-flatter test

Now make both hills a hundred times steeper, then a hundred times flatter, and keep every setting the same.

• A hundred times steeper: the shared step size of 0.015 blows up on the steep hill. After one step the dial is at 148, after two at −10,952.
• A hundred times flatter: the same shared step size crawls. The flat dial needs 24,591 steps.
• Adam (step size 0.1) takes 71 steps on the original hills, 71 on the steeper ones and 71 on the flatter ones, on exactly the same path every time.

That is a big part of why Adam is so popular: when a problem's scale changes, you don't have to hunt for a new step size.

The old way blows up or crawls for 24,591 steps; Adam takes the same path every time: 71 · 71 · 71.

Try it in PyTorch

In PyTorch, Adam is one line. A learning rate of 1e-3 is the usual starting point:

opt = torch.optim.Adam(model.parameters(), lr=1e-3)

Here are the two dials from this article, for one step:

import torch as t
w=t.nn.Parameter(-2*t.ones(2))
o=t.optim.Adam([w],lr=.1)
L=25*w[0]**2+w[1]**2/2
L.backward();o.step()
print(w.data)

It prints tensor([-1.9000, -1.9000]). Both dials moved exactly 0.1, even though one push was fifty times bigger. Change 25 to 2500 and run it again: still 0.1.

References

• Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. ICLR 2015. https://arxiv.org/abs/1412.6980
• CS878 Deep Learning, SEECS NUST, lectures 07–08 (course slides).

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

w ← w − η ∇L(w)

and nothing else. That makes it simple, but it also makes it forgetful: the moment the ground goes flat, it stops. Momentum fixes this with one extra number per weight, a velocity that remembers the slopes it has already rolled down. The examples below follow the CS878 lectures on gradient descent and scaling the optimization, and every number comes from running the update rule.

The trap

Take one weight and a loss with two dips:

L(w) = 0.12 (w − 2.5)² − 0.55 · exp(−(w + 1)² / 0.12)

Start at A, w = −3.5, where the loss is 4.32. There is a shallow dip B at w = −0.904 (loss 0.881) and the real minimum C at w = 2.5 (loss 0). Between them sits a bump only 0.13 higher than B.

Plain gradient descent with η = 0.1 takes big steps on the steep slope, smaller and smaller ones as it nears B, and by step 21 it has arrived. At B the gradient is zero, so every step after that is zero too. At step 50, at step 100, it is still in B, at a loss of 0.88.

Plain gradient descent stops in the shallow dip B: the gradient is zero there, so learning stops.

Add a velocity

Momentum keeps a velocity v and moves along it:

• v_t = β v_(t−1) + ∇L(w_t)
• w_(t+1) = w_t − η v_t

With β = 0.9, each step keeps 90% of the old velocity and adds the new gradient. (This is the form PyTorch uses. Lecture notes often write v = βv + (1 − β)∇L, which only rescales the learning rate.)

Run it from A with the same η = 0.1. By step 6 the ball is near B and its velocity is about 6. The gradient at B is zero, but the velocity isn't, so the ball keeps going: it clears the bump at step 7, overshoots all the way to w = 4.71, swings back without ever returning to the bump, and settles in C by step 89.

Momentum carries the ball over the bump; plain GD's ghost is still stuck in B.

What v remembers

Unroll the velocity and it becomes a weighted sum of every gradient so far:

v_t = ∇L_t + β ∇L_(t−1) + β² ∇L_(t−2) + …

The latest gradient counts fully, the one before 90%, then 81%, 73%, 66% and so on. The weights add up to 1/(1 − β) = 10, so momentum remembers about ten steps. With β = 0.1 they would be 1, 0.1, 0.01: it forgets almost at once. In practice β is usually around 0.9.

Eight bars, heights β^k, sum 1/(1 − β) = 10: gradients that agree stack toward 10; gradients that flip cancel, staying within ±0.53.

Agree or cancel

That memory does two different things depending on the gradients:

• When they agree (+1, +1, +1 …), they stack: v grows 1, 1.9, 2.71, 3.44 … toward 1/(1 − β) = 10. The step grows up to ten times.
• When they flip sign (+1, −1, +1 …), they cancel: v goes 1, −0.1, 0.91, −0.18, 0.84, −0.25 … and settles at about ±1/(1 + β) ≈ 0.53. The step stays small.

A narrow valley

Now two weights and a valley that is steep across and almost flat along: L = ½(w₁² + 50 w₂²), starting at (−4, 1), where the loss is 33. Count the steps until the loss drops below 0.001, with η = 0.015:

• Plain GD: 298 steps. A bigger rate doesn't help: at η = 0.045, each step multiplies the across direction by 1 − 0.045 · 50 = −1.25 and it blows up. So plain GD has to crawl along the floor.
• Momentum (β = 0.9): 73 steps. The along direction keeps agreeing, so it builds speed there. But the across direction overshoots and swings from side to side.

The narrow valley: plain GD crawls along the floor; momentum is faster but swings across.

Nesterov: look before you step

Nesterov momentum changes one thing: it takes the gradient where the velocity is about to carry the weight, not where the weight is now.

v_t = β v_(t−1) + ∇L(w_t − η β v_(t−1))

If that look-ahead point is already uphill, the gradient there points back, and the step brakes before it overshoots. On the same valley, Nesterov needs 18 steps, with almost no swing.

Nesterov looks ahead: the gradient at the look-ahead point brakes the swing. 298 · 73 · 18.

Try it in PyTorch

Every weight in a network gets its own velocity, with the same shape as the weight. In PyTorch it is one line:

opt = torch.optim.SGD(model.parameters(), lr=0.015,
                      momentum=0.9, nesterov=True)

And the velocity itself, with gradients that flip sign:

v, out = 0.0, []
for g in [1, -1, 1, -1, 1]:
    v = 0.9 * v + g
    out.append(round(v, 2))
print(out)

It prints [1.0, -0.1, 0.91, -0.18, 0.84]. Change every g to 1 and v climbs toward 10.

Key takeaways

• Plain gradient descent has no memory: it stops wherever the slope is zero, even in a shallow dip.
• Momentum keeps a velocity, v = βv + ∇L, and steps along it, so it can roll through flat spots and over small bumps.
• v is a sum of past gradients weighted by β^k, about 1/(1 − β) = 10 steps of memory at β = 0.9.
• Agreeing gradients stack (up to 10×); flipping ones cancel. That is why momentum speeds up along valleys.
• Nesterov takes the gradient at the look-ahead point, so it brakes before overshooting: 298 → 73 → 18 steps in the narrow valley.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Real training never does it that way. It takes a small random handful of examples, a mini-batch, and steps on that alone. This article shows why that works, what it costs, and how it looks in PyTorch. The numbers come from a tiny worked example and from the CS878 Deep Learning slides (lectures 07-08).

The gradient is an average

The loss of a model is an average over the training examples: L = (1/N) Σ ℓᵢ, where ℓᵢ is the loss on example i. A derivative of an average is the average of the derivatives, so the gradient is an average too: ∇L = (1/N) Σ ∇ℓᵢ.

Take a model with two weights, ŷ = w·x, a squared loss ℓᵢ = ½(ŷᵢ − yᵢ)², and four examples:

• x₁ = (4, −1), y₁ = 1
• x₂ = (0, 1), y₂ = 3
• x₃ = (1, 1), y₃ = 3
• x₄ = (1, −1), y₄ = 1

Start at w = (0, 0). There each example's gradient is −yᵢ·xᵢ, so the four arrows are (−4, 1), (0, −3), (−3, −3) and (−1, 1).

Put them tip to tail and they add up to (−8, −4). Divide by four and you get the true gradient, ∇L = (−2, −1). A step with learning rate ¼ moves the other way, to w = (½, ¼).

Four examples, four arrows, added tip to tail and divided by four: the true gradient (−2, −1).

Now make it a million examples. The slides use n = 2²⁰, about a million images. Every single step must touch all million. That is a lot of work for one small move.

A mini-batch is a cheap, honest guess

So take a shortcut. Shuffle the data, cut it into small batches, and average the gradients of one batch only:

g_B = (1/B) Σ over i in the batch of ∇ℓᵢ

With our four examples and B = 2, say the shuffle gives batches {1, 3} and {2, 4}.

• Batch one: ½[(−4, 1) + (−3, −3)] = (−7/2, −1). Too long.
• Batch two: ½[(0, −3) + (−1, 1)] = (−1/2, −1). Too short.

One-step guesses from them would land at (7/8, ¼) and (1/8, ¼), on either side of the true landing point (½, ¼). But their average is ½[(−7/2, −1) + (−1/2, −1)] = (−2, −1), exactly the true gradient.

That is the key property. Over the random choice of batch, the expected value of g_B is ∇L. The estimate is noisy, but it is unbiased: it does not lean in any direction.

Batch one's gradient is too long, batch two's too short; their average lands exactly on the true gradient.

How noisy is it?

If one example's gradient has a spread of σ around the true gradient, a batch of B examples has a spread of σ/√B (when the data set is much bigger than the batch).

That square root matters. Four times the batch costs four times the compute but only halves the noise. Going from 1 to 64 examples cuts the spread by a factor of 8. Past a point, a bigger batch buys very little extra accuracy for its cost. A small batch already points the right way.

The cloud of mini-batch gradients around the true one shrinks like σ/√B, while the cost grows like B.

A race on one budget

Give three methods the same budget: one pass over 1,024 examples, called one epoch, with the same learning rate.

• Full batch takes one careful step. After a whole epoch it is still far from the bottom.
• Pure SGD (B = 1) takes 1,024 shaky steps. It reaches the bottom but keeps jittering around it.
• Mini-batch (B = 64) takes 16 steady steps and ends near the bottom with only a small wobble.

There is a hardware reason too. A GPU computes many examples in parallel. A batch of 64 is one GPU call that keeps all its lanes busy, so the mini-batch epoch is 16 calls instead of 1,024.

If you plot the loss against iterations, as the lecture does, full batch is smooth, SGD is jagged, and mini-batch sits in between. That middle ground is why mini-batch SGD is the default.

One pass over 1,024 examples, all three at once: full batch takes 1 step, B = 64 takes 16, B = 1 takes 1,024; a B = 64 step lights all 64 GPU lanes together.

Try it in PyTorch

In PyTorch the batch size lives in the DataLoader. One batch is one iteration, one optimizer step. One full pass over the data is an epoch.

loader = DataLoader(data, batch_size=64, shuffle=True)
opt = optim.SGD(net.parameters(), lr=0.1)
for epoch in range(3):
    for xb, yb in loader:
        loss = loss_fn(net(xb), yb)
        opt.zero_grad()
        loss.backward()
        opt.step()

MNIST has 60,000 training images. In batches of 64 that is ⌈60,000 / 64⌉ = 938 iterations per epoch, and the last batch holds the 32 images left over. You can check it without any real data:

import torch.utils.data as D
import torch
X = torch.zeros(60000, 1)
dl = D.DataLoader(X, 64)
*_, last = dl
print(len(dl), len(last))

Output:

938 32

Change 64 to 1 and you get pure SGD: 60,000 steps per epoch. Change it to 60000 and you are back to full batch: one step per epoch. The optimizer is the same optim.SGD either way. The batches are what make it mini-batch SGD.

The PyTorch loop: one batch of 64 images flows through the network together, and 60,000 / 64 rounds up to 938 steps per epoch.

Noise can even help

At a flat saddle point the true gradient is almost zero, so full-batch descent barely moves. A mini-batch gradient keeps a little noise, and that is enough to shake the weights loose and let them roll downhill.

The noise does not fix everything. In a long, narrow valley the steps still zig-zag from wall to wall. Taming that zig-zag is the job of momentum, the next idea in this series.

Key takeaways

• The loss is an average over examples, so its gradient is an average of per-example gradients.
• A mini-batch gradient is a cheap estimate of the true gradient. It is noisy but unbiased: on average it is exactly right.
• Its noise shrinks like σ/√B, so four times the batch only halves the noise at four times the cost.
• In one epoch, mini-batches take many cheap steps that run in parallel on a GPU, between smooth full batch and jagged pure SGD.
• In PyTorch, batch_size in the DataLoader sets it: 60,000 images in batches of 64 is 938 steps per epoch.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

w ← w − η ∇L

The gradient comes from the model. The learning rate is a setting you choose, and it is the first one people tune. This article shows what it does on the smallest possible example, why there is a hard limit on how big it can be, and how practitioners pick it and change it during training. The ideas follow the CS878 Deep Learning lecture on learning rates.

One weight on a bowl

Take a single weight and the loss L(w) = w², a bowl with its bottom at w = 0. The gradient is 2w, so one step is

w ← w − η · 2w = (1 − 2η) · w

Start at w = 2, where the loss is 4, and try four learning rates:

• η = 0.01: each step multiplies w by 0.98. After ten steps, w has only reached 1.634 and the loss is still 2.67. It crawls.
• η = 0.4: each step multiplies w by 0.2. Two becomes 0.4, then 0.08, then 0.016. Fast.
• η = 0.9: each step multiplies w by −0.8. The weight jumps to the other side of the bowl each time (−1.6, 1.28, −1.024, 0.819), but the jumps shrink, so it still lands.
• η = 1.1: each step multiplies w by −1.2. The jumps grow: −2.4, 2.88, −3.456, 4.147. The loss climbs 5.76, 8.29, 11.94, 17.20. It explodes.

One weight on the bowl L = w², four learning rates: crawls, fast, zig-zags, explodes.

Why the edge is at one

Everything depends on the factor 1 − 2η. After t steps, w_t = (1 − 2η)^t · w₀.

• Below η = 0.5, the factor is between 0 and 1: the weight slides in without overshooting.
• At η = 0.5, the factor is 0: one jump lands exactly on the bottom.
• Between 0.5 and 1, the factor is between −1 and 0: the weight flips sides but shrinks.
• At η = 1, the factor is −1: it bounces between 2 and −2 forever.
• Past 1, the factor is below −1: it grows every step.

Nothing here is special to w². For any bowl L = (λ/2) w², with curvature λ (the second derivative), the factor is 1 − ηλ. So training is stable only if 0 < η < 2/λ, and one jump lands at η = 1/λ. For L = w², λ = 2: the edge is 1 and the one jump is 0.5.

The factor 1 − 2η decides everything: slides in below 0.5, one jump at 0.5, zig-zags up to 1, explodes past 1.

Millions of directions

A real network has millions of weights, so the loss has millions of directions, and they are not equally steep. Take two weights and L = ½(w₁² + 10 w₂²): one direction with curvature 1 (flat), one with curvature 10 (steep). Each step multiplies w₁ by (1 − η) and w₂ by (1 − 10η).

Starting from (4, 1):

• η = 0.02: the steep direction shrinks fast (×0.8 per step) but the flat one barely moves (×0.98). After ten steps the weights are at (3.27, 0.11): done on the steep side, stuck on the flat side.
• η = 0.21: 0.21 is past 2/10, the edge of the steep direction. w₂ is multiplied by −1.1 each step: 1, −1.1, 1.21, −1.331, 1.464. It explodes.
• η = 2/11 ≈ 0.18: the best single fixed rate, 2/(λ_min + λ_max). Both directions shrink by 9/11 per step, and after ten steps the weights are at (0.54, 0.13).

The steep direction sets the speed limit, η < 2/λ_max, and the flat direction then crawls. Even the best fixed rate shrinks the error by only (κ − 1)/(κ + 1) = 9/11 per step, where κ = λ_max/λ_min = 10 is how uneven the valley is.

The valley L = ½(w₁² + 10w₂²): the steep side sets the speed limit; the best fixed step is 2/11.

Picking a starting rate

Two quick checks get you most of the way:

1. Before training, look at the first loss. For softmax with C classes and small random weights, it should be about ln C, which is 2.30 for ten classes. If it isn't, something else is wrong.
2. Try a few learning rates a factor of ten apart: 1e-1, 1e-2, 1e-3, 1e-4. Run each for about a hundred iterations and keep the one whose loss drops fast without blowing up. Too small barely moves; too big turns into NaN.

Trying 1e-4 to 1e-1 from the same starting loss of 2.30: keep the one that drops fast.

Try it in PyTorch

The bowl example, at the learning rate that explodes:

w, lr = 2.0, 1.1
for t in range(4):
    w = w - lr * 2 * w
    print(round(w, 3))

It prints −2.4, 2.88, −3.456, 4.147. Change lr to 0.5 and w lands on 0 in one step.

In a real training loop the learning rate is one argument of the optimizer. A quick sweep tries a few values a factor of ten apart:

import torch
# model = your network
for lr in (1e-1, 1e-2, 1e-3, 1e-4):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    ...            # train ~100 iterations, keep the lr whose loss drops fast

Shrinking the rate during training (decay schedules) is its own episode.

Key takeaways

• The learning rate sets how far each gradient step goes; it is chosen, not learned.
• On a bowl with curvature λ, each step multiplies the error by 1 − ηλ: stable only if η < 2/λ, one jump at η = 1/λ.
• In a real network the steepest direction sets the speed limit, so the flat directions crawl.
• Start by checking the first loss (ln C), then try rates a factor of ten apart and keep the one that drops fast.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

In PyTorch you write the forward pass, call L.backward(), and every gradient is there. This article shows what happens in between. The engine that does it is called autograd, short for automatic differentiation.

We'll use the same network and the same numbers as last time, from the CS878 Deep Learning slides.

The forward pass builds the graph

Here is the network in PyTorch:

x  = torch.tensor(1.0)
w1 = torch.tensor(1/3, requires_grad=True)
w2 = torch.tensor(2.0, requires_grad=True)
y  = torch.tensor(0.0)
z  = x * w1
s  = z.relu()
yh = s * w2
L  = (yh - y)**2

The first four lines make the leaves of the graph: the input x, the target y and the two weights. Only the weights have requires_grad=True. That flag tells autograd we will want their gradients.

Every line after that runs one operation, and autograd records it. x * w1 makes a multiply node that holds 1/3. z.relu() adds a ReLU node, still 1/3. Multiplying by w2 gives 2/3, and the squared error gives the loss, 4/9.

You can see the record in what each result prints. z prints as tensor(0.3333, grad_fn=<MulBackward0>). The value is the number we expected. The grad_fn part is new: it is the function that made z.

Typing the forward pass builds the graph, one node per line: the loss is 4/9.

grad_fn links make the backward graph

Every result keeps its grad_fn, and every grad_fn keeps links to the functions that made its inputs. Follow those links from the loss and you have the whole graph, pointing backward.

One line of code can hold more than one operation. L = (yh - y)**2 is a subtraction and then a power, so PyTorch records two nodes: L.grad_fn is PowBackward0, and its input comes from SubBackward0.

The weights asked for gradients, so each gets an extra node at the very end of the backward graph, called AccumulateGrad. Think of it as a bucket: it is where the weight's .grad will be stored. The input x and the target y did not ask for gradients, so autograd skips them.

The backward graph: every operation is now its backward function, the arrows point down, and each weight has an AccumulateGrad bucket.

L.backward() walks the graph in reverse

Calling L.backward() starts with a gradient of 1 at the loss and visits the nodes in reverse order. Each node knows one small rule: take the gradient from above and multiply it by its own local derivative.

• Pow: the local derivative of d² is 2d, so d gets 1 × 2d = 4/3.
• Sub: the local derivative of ŷ − y with respect to ŷ is 1, so ŷ gets 4/3. The path to y is skipped.
• Mul: a multiply node swaps values. w2 gets 4/3 × s = 4/9, and s gets 4/3 × w2 = 8/3.
• ReLU: z was positive, so the gate is open and passes 8/3 through.
• Mul: w1 gets 8/3 × x = 8/3. The path to x is skipped.

The buckets fill: w1.grad is tensor(2.6667), or 8/3, and w2.grad is tensor(0.4444), or 4/9. Those are the numbers we found by hand.

After L.backward(): the buckets hold 8/3 and 4/9, the same numbers as the hand-worked backprop.

When a value is used twice, gradients add

Take y = w * w with w = 3. The multiply node has two inputs, and both are w. On the way back, each path sends 3 (the other input's value). The two contributions meet in w's bucket and add up to 6, which is the derivative of w², 2w. This is the multivariable chain rule: when a value feeds several paths, its gradient is the sum over the paths.

The bucket keeps adding across calls too. Run the forward and backward pass again and w.grad goes from 6 to 12. That's why every training loop calls optimizer.zero_grad() (or sets the gradients to None) before each backward pass.

Running backward twice: the bucket jumps from 6 to 12, so training loops call zero_grad.

Why go backward?

A network has one loss and millions of weights. You could also push derivatives forward, from one weight up to the loss. That's forward-mode differentiation, and it needs one pass per weight: a million weights, a million passes.

Reverse mode, the one autograd uses, starts from the single loss and reaches every weight in one sweep. It costs about as much as a couple of forward passes, whatever the number of weights. With one output and many inputs, going backward is the only sensible direction.

Reverse mode: one wavefront sweeps down from the loss through every layer and reaches all 36 weights in a single pass.

The graph is rebuilt every step

After backward(), PyTorch frees the graph. The next forward pass records a new one. Because the graph is built while your code runs, ordinary Python works inside a model: an if can add a node on one step and leave it out on the next, and a loop can run a different number of times.

The same code records two different graphs: with s > 0 the if adds a ×2 node.

Try it in PyTorch

import torch
w = torch.tensor(3.,
  requires_grad=True)
(w * w).backward()
(w * w).backward()
print(w.grad)

It prints tensor(12.), not 6, because the second backward pass adds to the first. Add w.grad = None between the two backward calls and it prints tensor(6.).

Key takeaways

• Autograd records every operation of the forward pass as a graph. Each result's grad_fn is the function that made it.
• L.backward() walks that graph in reverse. Each node multiplies the gradient from above by its local derivative.
• Weights with requires_grad=True collect their gradient in .grad through AccumulateGrad. Tensors without the flag are skipped.
• Gradients from several paths add up, and .grad keeps adding across steps, so training loops call zero_grad().
• Reverse mode gets every gradient in one sweep, and the graph is rebuilt on every forward pass, so plain Python control flow works.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

That's what backpropagation does. After the network makes its prediction (the forward pass), it sends the error back through the network (the backward pass). On the way, every weight picks up a number, its gradient, that says how much the loss would change if that weight moved a little. That number is the weight's share of the blame.

The examples below are the ones from the CS878 Deep Learning lecture slides, with the same numbers.

A tiny graph, forward

Start with something small enough to see all at once:

f(x, y, z) = (x + y) · z

Draw it as a graph with two steps. An add node computes d = x + y, and a multiply node computes f = d · z.

Put in x = 1, y = −3 and z = 4. The add node gives d = 1 + (−3) = −2. The multiply node gives f = −2 · 4 = −8. That's the forward pass: values flow from the inputs to the output.

The forward pass. The add node turns 1 and −3 into −2, and the multiply node turns −2 and 4 into −8.

The same graph, backward

Now go the other way. We want to know how f changes when each input changes: ∂f/∂x, ∂f/∂y and ∂f/∂z.

Start at the output with a gradient of 1, because f changes exactly as much as f does.

A multiply node swaps: each input gets the upstream gradient times the other input's value. For f = d · z, the local gradients are ∂f/∂z = d and ∂f/∂d = z. So z gets 1 × (−2) = −2 (the value of d) and the sum gets 1 × 4 = 4 (the value of z).

An add node copies: it passes the gradient straight through to every input. For d = x + y, both local gradients are 1, so x and y each get 4 × 1 = 4.

Check one of them with the chain rule: ∂f/∂x = ∂f/∂d · ∂d/∂x = 4 · 1 = 4.

The backward pass. Starting from 1 at the output, the multiply node sends −2 to z and 4 to the sum; the add node passes 4 to both x and y.

Every node does the same job

Look at what each node did. It took the gradient arriving from above, the upstream gradient, and multiplied it by its own local gradient. For any node, with in one of its inputs and out its output:

∂L/∂in = ∂L/∂out · ∂out/∂in (upstream times local)

That's the whole algorithm. Each node only needs to know its own small derivative. It never has to see the rest of the network. The chain rule from the last episode, applied node by node, from the output back to the inputs.

A real network

Now a tiny network: one input x = 1, a first weight w₁ = 1/3, a ReLU, a second weight w₂ = 2, and a squared-error loss against a target of y = 0.

Forward:

• z = x · w₁ = 1/3
• s = max(0, z) = 1/3 (the ReLU's output; σ usually means the sigmoid)
• ŷ = s · w₂ = 2/3
• L = (ŷ − y)² = 4/9

The network predicts 2/3 when the answer is 0. That's the mistake, and the loss of 4/9 measures it.

The network after its forward pass. It predicts 2/3 instead of 0, a loss of 4/9.

Handing out the blame

Backward, from the loss, one node at a time:

• At the output: ∂L/∂ŷ = 2(ŷ − y) = 4/3.
• The multiply node ŷ = s · w₂ swaps. The second weight gets 4/3 × s = 4/3 × 1/3 = 4/9, and the other branch gets 4/3 × w₂ = 4/3 × 2 = 8/3.
• The ReLU's input was positive (z = 1/3), so the gate is open and its local gradient is 1: z gets 8/3 × 1 = 8/3. Had z been negative, the gate would be shut and pass back nothing.
• The multiply node z = x · w₁ swaps too. The first weight gets 8/3 × x = 8/3 × 1 = 8/3.

Multiply the factors along the path and you get the slides' chain rule, ∂L/∂w₁ = ∂L/∂ŷ · ∂ŷ/∂s · ∂s/∂z · ∂z/∂w₁ = 4/3 · 2 · 1 · 1.

So the first weight gets six times as much of the blame as the second: 8/3 = 6 × 4/9. One backward pass found both numbers.

The backward pass through the network. The first weight's gradient is 8/3, the second weight's is 4/9.

One step downhill

Gradient descent then moves every weight at once, each against its own gradient, scaled by a learning rate α:

w ← w − α · ∂L/∂w

With α = 0.05, the first weight moves six times as far as the second: w₁ goes from 1/3 to 1/3 − 0.05 · 8/3 = 0.20, and w₂ from 2 to 2 − 0.05 · 4/9 ≈ 1.98. Run the forward pass again with the new weights: z = 0.20, s = 0.20, ŷ ≈ 0.40, and the loss drops from 0.44 to about 0.16.

One gradient step with α = 0.05. The weights become 0.20 and 1.98, and the loss falls from 0.44 to 0.16.

Why it matters

A real network has millions of weights. Nudging them one at a time would take millions of forward passes per step. Backpropagation gets all of them in one backward pass that costs about as much as the forward pass. It does this by reusing work: the 4/3 at the output was computed once and shared by both weights.

It became the standard way to train neural networks after Rumelhart, Hinton and Williams showed in 1986 that it lets hidden layers learn useful features.

Try it in PyTorch

PyTorch runs backpropagation for you. Build the loss, call backward(), and every weight's gradient appears in .grad:

import torch
w = torch.tensor([1/3, 2.],
  requires_grad=True)
L = (w[1] * w[0].relu())**2
L.backward()
print(w.grad)

It prints tensor([2.6667, 0.4444]): 8/3 and 4/9, the same numbers we found by hand. (Here x = 1 and y = 0, so they drop out of the formula.) Change 1/3 to −1/3 and both gradients become zero: the ReLU outputs 0, so no blame gets through.

Key takeaways

• The forward pass computes values; the backward pass sends gradients from the loss back to every weight.
• Each node multiplies the gradient from above by its own local gradient.
• A multiply node swaps: each input gets the gradient times the other's value. An add node copies: it passes the gradient straight through. An open ReLU multiplies by 1.
• In the slides' network, one backward pass gives the first weight 8/3 and the second 4/9, and one step with α = 0.05 takes the loss from 0.44 to 0.16.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Training keeps asking one question: if I nudge this weight a little, how much does the loss move? For a weight ten layers down, the nudge has to travel through every layer above it before it reaches the loss. The chain rule tells you how big it is when it arrives.

A network is a chain

Written as math, a ten-layer network is a function inside a function inside a function:

f₁₀( ⋯ f₂( f₁(x) ) ⋯ )

Picture it as a tower of dials on one shaft: the input at the bottom, a dial for each layer above it. Turn the bottom dial a little and every dial above it turns too, but not by the same amount. Each layer has its own ratio, which can make the turn bigger or smaller.

A network as a tower of dials. Turn the input at the bottom and every layer above it turns; the blue wedges show how far each one moved.

Two links, real numbers

Take just two layers. The first triples its input, y = 3x, and the second squares that, z = y². Start the input at x = 1: then y = 3 and z = 9.

Now raise the input by 0.01, to x = 1.01. Then y = 3.03: it moved 0.03, three times the nudge, so its ratio is 3.

And z = 3.03² = 9.1809: it moved about 0.18, six times y's own move of 0.03, so its ratio is 6.

Why six? Write out (3 + h)² = 9 + 6h + h². For a tiny h, the h² hardly matters, so the change is about 6h. That ratio is the slope of y² at y = 3: its derivative.

Close on two links. Raising the input by 0.01 moves the first layer by 0.03 (ratio 3) and the second by 0.18 (ratio 6).

The ratios multiply

The first link multiplies the nudge by 3 and the second by 6, so together it's 3 × 6 = 18. The real numbers agree: z moved 0.18 for a nudge of 0.01, which is 18 times.

That's the chain rule. Written with derivatives:

dz/dx = dz/dy · dy/dx = 6 × 3 = 18

Each layer only needs to know its own slope: find each link's ratio where it stands, then multiply.

You've seen this on a clock

A clock is a chain too. The minute hand turns 12 times as fast as the hour hand, and the second hand turns 60 times as fast as the minute hand. So the second hand turns 12 × 60 = 720 times as fast as the hour hand: you multiply the ratios along the chain, exactly as with the layers.

Ten layers, ten ratios

A network with ten layers multiplies ten ratios, and a product of many numbers moves fast in both directions.

• If every layer halves the turn, the top moves 0.5¹⁰ ≈ 0.001 as far as the input. The nudge has all but vanished.
• If every layer doubles it, the top moves 2¹⁰ = 1024 times as far. The nudge has exploded.

Ten layers that each double the turn. The bottom barely moves, and the top spins 1024 times as far.

This is why very deep networks were once so hard to train: the signal about how a deep weight affects the loss either faded away or grew out of control.

The same rule in a real network

Here is a small three-layer network: two inputs, three hidden neurons with ReLU, and one output, scored by squared error against a target of 1. With inputs 2 and 1, the hidden neurons come out at 1, 0.5 and 0.2, the output is 3, and the loss is (3 − 1)² = 4.

Pick one weight, w = 0.5, from the first input to the top hidden neuron, and follow its path to the loss. It has three links:

• the hidden neuron moves 2 times as much as w (the input it multiplies is 2);
• the output moves 3 times as much as that hidden neuron (its weight is 3);
• the loss moves 4 times as much as the output (the slope of (y − 1)² at y = 3).

So ∂L/∂w = 2 × 3 × 4 = 24. Check it: raise w from 0.50 to 0.51, and the loss goes from 4.00 to 4.24.

A three-layer network. One weight's path to the loss has ratios 2, 3 and 4, so the loss moves 24 times as much as the weight.

Try it in PyTorch

PyTorch applies the chain rule for you. torch.func.grad returns a function's derivative, even when the function is a chain of other functions:

from torch import tensor, func
y = lambda x: 3 * x
z = lambda y: y ** 2
f = lambda x: z(y(x))
x = tensor(1.)
print(func.grad(f)(x))

It prints tensor(18.), the same 3 × 6 we found by hand. Add a third layer that halves its input, and the gradient of the longer chain should come out at 9.

Key takeaways

• A network is a chain of layers, and a nudge at the bottom ripples through every layer above it.
• Each layer scales the nudge by its own ratio: its local slope, the derivative.
• The chain rule: multiply the ratios along the chain. dz/dx = dz/dy · dy/dx.
• With many layers the product can shrink to almost nothing (0.5¹⁰ ≈ 0.001) or blow up (2¹⁰ = 1024).
• In a real network, a weight's effect on the loss is the product of the ratios along its path: 2 × 3 × 4 = 24.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

That's enough. If the ground tilts down to your left, step left, then feel again. Repeat until the ground is flat. That loop is gradient descent, and it's how almost every neural network is trained.

The ground is the loss

A network in training is in the same position. Its loss, one number that says how wrong its answers are, depends on its weights: change a weight and the loss goes up or down.

Picture a network with just two weights. For every pair of weights you can compute the loss, so you can draw it as a height over a flat map of the two weights. That gives a landscape. Where you stand on the map is the current weights. The height under your feet is the current loss. Training means finding the lowest point.

In our example the landscape is a long valley:

loss = (w₁ − 3)² + 3 · (w₂ − 1)²

The bottom is at w₁ = 3 and w₂ = 1, where the loss is 0. We start at w₁ = 0 and w₂ = 3, where the loss is 21.

The valley the network can't see. The ball is the current weights; the blue patch is the only ground it can feel.

The network never sees this picture. A real network has millions of weights, so its landscape can't be drawn at all. Like the hiker, it only knows the ground where it stands.

The slope is the gradient

The thing you feel under your feet has a name: the gradient. It's the slope of the loss in every weight's direction at once, and it points uphill, the way the loss grows fastest.

At our starting point the gradient is (−6, 12). The 12 says the loss climbs steeply if weight 2 grows. The −6 says it also climbs if weight 1 shrinks. Going uphill would be a bad idea, so gradient descent does the opposite and steps against the gradient, downhill.

The gradient points uphill (grey arrow). The step goes the other way, downhill (blue).

How big is a step?

Each step is the gradient times a small number called the learning rate, written η (eta):

w ← w − η · ∇L

With a learning rate of 0.1, the first step moves the weights by 0.1 × (−6, 12) = (−0.6, 1.2) the other way. They go from (0, 3) to (0.6, 1.8), and the loss drops from 21 to 7.7.

Then you feel the slope again and take the next step. Here is what twenty steps do:

• after 1 step, the loss is 7.7;
• after 2 steps, 4.0;
• after 5 steps, 0.97;
• after 10 steps, 0.10;
• after 20 steps, 0.001, almost zero.

Notice that the steps get shorter by themselves. Near the bottom the ground flattens, so the gradient shrinks, and a small gradient times the learning rate makes a small step. Nobody tells the network to slow down near the minimum. The slope does it.

Twenty real steps at learning rate 0.1. The loss goes from 21 to almost zero, and the steps shrink as the ground flattens.

The learning rate is your stride

The gradient gives the direction, and the learning rate decides how far you go each time. To see why that matters, look along the valley floor from the side, where it's a simple curve, loss = (w − 3)², starting at w = 0 with a loss of 9.

• Learning rate 0.1: the hops shrink as the curve flattens, and the ball settles at the bottom.
• Learning rate 0.01: tiny steps. After twenty of them the loss is still 4.0. You're walking the right way, just very slowly.
• Learning rate 1.1: giant leaps. Every step overshoots the bottom and lands higher on the other side. Four steps later the loss has grown from 9 to 38.7, and it keeps growing.

The valley floor from the side. At 1.1 every step leaps over the bottom and climbs (grey arcs); at 0.1 the hops shrink and the ball settles at the bottom (blue).

There's a simple reason for the blow-up. On this curve, each step multiplies your distance from the bottom by (1 − 2η). With η = 0.1 that's 0.8, so you get 20% closer every step. With η = 1.1 it's −1.2: you land on the other side, 20% farther away than before.

That's why the learning rate is usually the first setting people tune. A common recipe from the lecture notes: try 0.1, 0.01, 0.001 and 0.0001, and keep the one that makes the loss drop fast within the first hundred steps.

Try it in PyTorch

PyTorch can compute the gradient for you. torch.func.grad turns a function into its slope:

from torch import tensor, func
L = lambda w: (w - 3) ** 2
w = tensor(0.)
for _ in range(20):
  w -= 0.1 * func.grad(L)(w)
print(w)

It prints tensor(2.9654): twenty steps from 0 get within 0.035 of the bottom at 3. Change 0.1 to 1.1 and run it again to watch the steps leap out of the valley.

Key takeaways

• Training is a walk down the loss landscape, where your position is the weights and your height is the loss.
• The gradient is the slope under your feet. It points uphill, so each step goes against it: w ← w − η · ∇L.
• Steps shrink by themselves near the bottom, because the slope flattens.
• The learning rate is your stride. Too small and you crawl, too big and you leap out of the valley.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

One number, one log

Cross-entropy looks at a single probability: the one the model gave to the right answer. Call it p. The loss is minus its log:

loss = −log(p)

If the picture is a cat and the model said 67% cat, the loss is −log(0.67) ≈ 0.4. If it said 1% cat, the loss is −log(0.01) ≈ 4.6.

The loss as a landscape

Take a classifier with three classes, and hold the bird score still. Two numbers are left: the cat score and the dog score. For every pair, cross-entropy gives a loss. Make that loss the height, and you get a landscape.

The loss over the cat and dog scores: a landscape. The ball is a confidently wrong model: cat 1%, loss 4.61.

Where the cat score wins, the ground is flat and close to zero. A model that's confident and correct sits there at almost no cost: at 98% cat, the loss is 0.02.

Where the cat score wins, the ground is flat and close to zero.

Where the dog score wins, the ground climbs, and it never levels off. Far enough up that ramp the loss grows by one for every extra point of dog score. There is no flat spot for a confident mistake.

Where the dog score wins, the ground keeps climbing.

Training rolls downhill

Training follows the slope. For softmax with cross-entropy, the slope with respect to each score is just the prediction minus the truth, where the truth is 1 for the right class and 0 for the others. So each training step moves the scores by that amount, times a learning rate:

z ← z − (p − y)

Start at the confident mistake, cat 1% and dog 99%, with a learning rate of 1. Three steps take the cat from 1% to 7%, to 32%, to 64%, and the loss falls from 4.61 to 0.44. The ramp is steep exactly where the model is most wrong, so the push back is strongest there.

Three real gradient-descent steps from the top of the ramp: cat 1% → 64%.

The same loss as a bet

Here's another way to see it. Think of the model's probabilities as a bet: 100 chips spread across the answers, 67 on cat, 24 on dog and 9 on bird. When the answer turns out to be a cat, cross-entropy sends a bill: minus the log of the cat's share, −log(67/100) ≈ 0.4.

The model bets 100 chips. It's a cat, so the bill is −log(0.67) ≈ 0.4.

Halve the chips on the truth, and the bill grows by the same amount every time, log(2) ≈ 0.69:

• 67 chips cost 0.40;
• 34 chips cost 1.09;
• 17 chips cost 1.79;
• one chip costs 4.61;
• zero chips: there is no limit.

Every halving adds 0.69. One chip costs 4.6; at zero, the bill has no limit.

That's why a confident mistake costs so much: being sure of the wrong answer means betting almost nothing on the truth, and every halving of that bet keeps adding to the bill.

Try it in PyTorch

You never write the log yourself. nn.CrossEntropyLoss takes the raw scores and the right class and does the softmax and the log for you:

import torch
from torch import nn
z = torch.tensor([[2., 1, 0]])
y = torch.tensor([0])  # cat
print(nn.CrossEntropyLoss()(z, y))

It prints tensor(0.4076), the 0.4 from above. Pass it probabilities instead of raw scores and it applies softmax a second time, and the loss comes out wrong.

Key takeaways

• Cross-entropy = −log(probability of the right answer).
• Over the scores it is a landscape: flat where the model is right, a ramp with no top where it is confidently wrong.
• Its slope is prediction minus truth, so training pushes hardest where the model is most wrong.
• As a bet: every halving of the chips on the truth adds 0.69, and zero chips has no limit.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

That measurement is the loss function. It turns all of a network's errors into a single number, and training is the process of pushing that number down.

One number for "how wrong"

A loss function compares the network's predictions with the right answers and returns one number:

• Zero means every prediction is perfect.
• Bigger means worse.

That's it. Everything a network learns, it learns by trying to make this number smaller.

A simple example

Say a network draws a line to predict three data points. It doesn't fit perfectly. Its predictions miss the three points by 3, 2, and 1.

How bad is that, in one number? It depends on which loss function you pick.

The network's line misses three points by 3, 2 and 1.

Squared loss

The squared loss squares each miss, then adds them up:

• 3² = 9
• 2² = 4
• 1² = 1

The total is 9 + 4 + 1 = 14. When you average it over all the data points, this is called the mean squared error, or MSE. Here, that's 14 ÷ 3 ≈ 4.7.

Squaring makes every miss positive, and it punishes big misses much more than small ones. A miss of 3 costs nine times as much as a miss of 1.

Squared loss: the misses 3, 2 and 1 cost 9, 4 and 1, for a total of 14.

Absolute loss

The absolute loss is simpler. It just adds up the size of each miss, ignoring whether the prediction was too high or too low:

3 + 2 + 1 = 6.

Averaged over the data points, this is called the mean absolute error, or MAE. Here, that's 6 ÷ 3 = 2.

What one outlier does

Now a new point arrives, and the network misses it by 5.

• Squared, that one miss costs 5² = 25. That's more than all the other misses combined (14).
• The absolute loss only adds 5.

The totals jump from 14 to 39 for squared loss, but only from 6 to 11 for absolute loss.

So one unusual point can dominate the squared loss. If your data has a few wild outliers, a network trained with squared loss will bend over backwards to reduce them, sometimes at the expense of fitting everything else.

One new point misses by 5. Squared, it costs 25, more than the other three misses combined. Absolute only adds 5.

How training uses the loss

Training adjusts the network's weights to push the loss down, one small step at a time. Each step asks: "If I nudge this weight a little, does the loss go up or down?" Then it moves the weight in the direction that lowers the loss.

Which loss you choose shapes what the network cares about. Squared loss cares a lot about big mistakes. Absolute loss treats every unit of error the same.

Training nudges the weights to push the loss down, one small step at a time.

Quick check

Your data has a few wild outliers. Which loss stays calmer: squared, or absolute?

Answer: absolute. It grows in step with each miss, while squaring blows big misses up.

Key takeaways

• A loss function turns all of a network's errors into one number. Zero is perfect.
• Squared loss squares each miss (3, 2, 1 → 9 + 4 + 1 = 14). Averaged, it's the mean squared error.
• Absolute loss adds the miss sizes (3 + 2 + 1 = 6). Averaged, it's the mean absolute error.
• Squared loss is very sensitive to outliers: a single miss of 5 costs 25.
• Training adjusts the weights, step by step, to push the loss down.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Softmax is the small function that turns them into probabilities. It takes two steps, and you can do both by hand.

Raw scores are not probabilities

Say a network has three classes: cat, dog, and bird. For one image, its last layer gives these raw scores (often called logits):

• Cat: 2
• Dog: 1
• Bird: 0

Cat clearly scores highest. But these numbers have two problems if you want to read them as probabilities. They can be negative, and they don't add up to 1. A score of 2 doesn't tell you "how sure" the network is.

The network's last layer gives one raw score per class: cat 2, dog 1, bird 0.

Step 1: take the exponential

First, softmax raises the number e (about 2.718) to the power of each score. This is called taking the exponential.

• Cat: e² ≈ 7.4
• Dog: e¹ ≈ 2.7
• Bird: e⁰ = 1

The exponential has a useful property: its result is always positive, even when the score is negative. A score of −3 becomes e⁻³ ≈ 0.05, small but still positive. It also stretches the gaps, so higher scores pull further ahead.

Step 1: the exponential turns 2, 1 and 0 into about 7.4, 2.7 and 1. Every result is positive.

Step 2: divide by the total

Next, add the three results: 7.4 + 2.7 + 1 ≈ 11.1. Then divide each one by that total.

• Cat: 7.4 ÷ 11.1 ≈ 67%
• Dog: 2.7 ÷ 11.1 ≈ 24%
• Bird: 1 ÷ 11.1 ≈ 9%

Together, they add up to 100%. Now the numbers are real probabilities: all positive, and they sum to one.

As a formula, for a class with score zᵢ: softmax(zᵢ) = e^(zᵢ) ÷ (the sum of e^(z) over all classes).

Step 2: divide each exponential by the total. Cat gets 67%, dog 24%, and bird 9%, adding up to 100%.

Why "soft" max?

A hard max would simply pick the winner and give it everything: cat 100%, dog 0%, bird 0%. That throws away information about how close the race was.

Softmax is the soft version. The winner gets the biggest share, but the others keep a share too. That matters during training, because the network can learn from how confident it was, not just whether it was right.

A hard max would give cat everything and the others nothing. Softmax keeps a share for every class.

Quick check

Add 10 to every score: cat 12, dog 11, bird 10. Do the probabilities stay the same, or does cat's share grow?

Answer: they stay the same. Adding 10 multiplies every exponential by the same amount, e¹⁰. That common factor cancels when you divide by the total. Only the differences between scores matter.

Key takeaways

• A classifier's last layer outputs raw scores, which can be negative and don't sum to one.
• Softmax takes the exponential of each score, which makes every value positive.
• Then it divides by the total, so the results add up to 100%.
• Scores of 2, 1 and 0 become about 67%, 24% and 9%.
• Shifting every score by the same amount doesn't change the result. Only the gaps matter.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

That ingredient is the activation function. Here's why it matters, in a few lines of algebra.

What a layer does

Every layer in a neural network does two things:

1. It multiplies and adds: each input is multiplied by a weight, the results are summed, and a bias is added.
2. It passes the result through an activation function: a small bend.

The first step is simple arithmetic. The second step is the one that's easy to overlook, and it's the one that makes depth worth having.

What happens without the bend

Let's remove the activation function and watch what happens with two tiny layers.

• Layer one doubles its input and adds one: h = 2x + 1.
• Layer two triples that and subtracts four: y = 3h − 4.

Now put them together. Substitute the first into the second: y = 3(2x + 1) − 4 = 6x − 1.

Two layers, and the result is a single multiply-and-add. It's still just one straight line.

Two layers without a bend: 2x + 1, then 3h − 4. Combined, they collapse into a single line, 6x − 1.

Stacking doesn't help

This isn't a coincidence of the numbers we picked. Any multiply-and-add fed into another multiply-and-add is still a multiply-and-add.

So stack a hundred of these layers, and they still collapse into one. All that depth, and the network can only draw a straight line.

Add the bend: ReLU

Now add an activation function. The most popular one is called ReLU (rectified linear unit). Its rule is:

• Negative numbers become zero.
• Positive numbers pass straight through.

So −3 becomes 0, and 2 stays 2. That's it. It looks almost too simple to matter.

ReLU: negative inputs become zero, positive inputs pass straight through.

From bends to curves

That one kink at zero is enough to stop the collapse. The layers can't be merged into a single line anymore.

Better still, bends can be combined. Add two ReLUs, one of them flipped, and you get a peak. Combine many of them, with different positions and slopes, and they trace a curve, one straight piece at a time.

This is how networks build complicated functions out of very simple parts.

Many ReLU pieces, each a straight segment with one bend, combine to trace a curve.

Sigmoid vs ReLU

Older networks often used a smooth S-shaped curve called sigmoid, which squashes any number into the range between 0 and 1.

Today, ReLU is the standard choice in most networks, because it's simple and fast to compute. It also tends to make deep networks easier to train.

Sigmoid, a smooth S-curve, was common in older networks. ReLU is the standard choice today.

Quick check

A network has ten layers, with no activation functions. Can it draw a curve, or only a straight line?

Answer: only a straight line. Without a bend, all ten layers collapse into one multiply-and-add.

Key takeaways

• Every layer multiplies, adds, and then applies an activation function.
• Without the activation, any stack of layers collapses into a single straight line, like 2x + 1 and 3h − 4 becoming 6x − 1.
• ReLU turns negatives into zero and passes positives through. That one bend stops the collapse.
• Many bends combined can trace curves.
• Sigmoid was common in older networks. ReLU is the default today.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

The surprising answer: from straight lines, lots of them, combined by a hidden layer.

One line can't fence in a cluster

A neuron splits the plane in two with a straight line. Points on one side get a "yes," and points on the other side get a "no."

That works when the classes sit on opposite sides of the page. But look at this case: the blue points form a cluster in the middle, surrounded by the others. Wherever you draw one straight line, it can't fence the cluster in.

The blue points sit in the middle. A single straight line can't separate them from the points around them.

Add a hidden layer

So add a layer of neurons between the inputs and the output. This is a hidden layer. Give it three neurons.

Each hidden neuron draws its own line, and each one asks a single question about any point: "Is this point on my side?" The answer is yes (1) or no (0).

The output neuron asks for agreement

The output neuron adds up the three answers. It only fires when the sum passes 2.5, so it needs all three to say yes.

The only place where all three lines agree is the region between them. Together, the three lines fence off a triangle.

Three hidden neurons draw three lines. The output fires only where all three agree, which is a triangle.

A worked example

Take a point inside the triangle. All three neurons say yes, so the sum is 1 + 1 + 1 = 3. That passes 2.5, so the output fires.

Now take a point just past one of the lines. Only two neurons say yes, so the sum is 0 + 1 + 1 = 2. That's below 2.5, so the output stays off.

That threshold turns the output neuron into an "AND" of all the hidden neurons.

More neurons, rounder shapes

The triangle is a rough fit, and its corners catch a few outsiders. The fix is simple: use more hidden neurons.

• With six neurons, you get six lines and a tighter hexagon.
• With twenty, the edge looks almost like a circle.

The edge is never truly curved. It's made of many short straight pieces, like a polygon with lots of sides. Add enough of them, and the difference stops mattering.

More hidden neurons means more straight edges. With twenty, the boundary looks almost like a circle.

Go deeper

A second hidden layer can combine the shapes from the first one. That lets a network fence off things a single shape can't, like two separate islands, or a ring with a hole in the middle.

This is why depth matters. Each layer builds on the shapes from the layer before it.

A second hidden layer combines shapes: two islands, or a ring with a hole.

Quick check

A hidden layer has five neurons. Does the output fence off a pentagon, or a perfect circle?

Answer: a pentagon. Five neurons draw five straight lines, so the edge has (at most) five sides. A perfect circle would need infinitely many.

Key takeaways

• One neuron draws one straight line.
• A hidden layer draws many lines, and the output neuron fires only where they agree.
• Three lines make a triangle, six a hexagon, and twenty something close to a circle.
• More layers combine shapes into islands, rings, and other complex regions.
• Hidden layers never bend a line. They join many straight ones into a curve.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

Understanding why takes one picture. And the fix explains why every modern neural network has hidden layers.

What XOR does

XOR stands for "exclusive or." It outputs 1 when exactly one of its two inputs is 1.

• 0 and 0 give 0.
• 0 and 1 give 1.
• 1 and 0 give 1.
• 1 and 1 give 0.

So it's "one or the other, but not both."

The XOR truth table. The output is 1 only when the two inputs differ.

Put the four cases on a grid

Now plot each case as a point, with the first input across and the second input up. Color the points by their answer: blue for 1, hollow for 0.

The two 1s sit on one diagonal. The two 0s sit on the other diagonal.

One neuron, one straight line

A single neuron (a perceptron) decides by drawing one straight line. Everything on one side gets a 1, and everything on the other side gets a 0.

Try to place that line for XOR:

• A vertical line catches one of the 1s, but also a 0.
• Turn it into a diagonal, and it catches both 1s, but still grabs a 0.

Wherever you put it, the line mixes the classes. No straight line can separate them. Problems where one line does work are called "linearly separable." XOR is not.

However you draw a single straight line, it puts a 0 on the same side as the 1s.

The book that cooled the field

In 1969, Marvin Minsky and Seymour Papert proved this limit in their book Perceptrons. They showed that single-layer perceptrons can't compute functions like XOR.

The result helped cool interest in neural networks for years. Research slowed until people had practical ways to train networks with more than one layer.

The fix: a hidden layer

The answer is to add a layer of neurons between the inputs and the output. It's called a hidden layer because you never see its values directly.

For XOR, two hidden neurons are enough:

1. The first neuron asks: is at least one input on? That's OR.
2. The second neuron asks: are they not both on? That's NAND ("not and").
3. The output neuron fires only when both answers are yes. That's AND.

Check it: for inputs 1 and 0, OR says yes and NAND says yes, so the output is 1. For 1 and 1, OR says yes but NAND says no, so the output is 0.

A hidden layer solves XOR: OR and NAND feed an AND output.

Two lines make a band

Here's what that looks like on the grid. Each hidden neuron draws its own straight line. Together, the two lines fence off a diagonal band, and both 1s land inside it.

One neuron gives you one line. A layer of neurons gives you many lines, and the next neuron can combine them into shapes. That's the core reason deep networks have layers.

The OR line and the NAND line fence off a band that holds both 1s and neither 0.

Quick check

One neuron draws one straight line. Which can it learn: the AND gate, or the XOR gate?

Answer: AND. AND outputs 1 only for the input (1, 1). That single point sits in a corner, so one straight line can cut it off. XOR needs two.

Key takeaways

• XOR outputs 1 when exactly one input is 1.
• A single neuron draws one straight line, and no straight line separates XOR's 1s from its 0s.
• Minsky and Papert proved this limit in 1969, which slowed neural network research for years.
• A hidden layer fixes it: OR and NAND draw two lines, and AND combines them into a band.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.

It couldn't do any of that. But it introduced an idea that every modern neural network still uses: a machine that improves by fixing its own mistakes.

How a perceptron makes a decision

Under the hood, a perceptron is a single artificial neuron. It does three things:

1. It takes some inputs, like x₁ and x₂.
2. It multiplies each input by a weight (w₁ and w₂) and adds them up.
3. It compares the sum with a threshold, written θ (theta).

If the sum clears the threshold, the perceptron outputs 1. Otherwise, it outputs 0. That's the whole decision: yes or no.

A perceptron: weighted inputs are summed, then a threshold step turns the sum into a 1 or a 0.

The rule draws a straight line

Here's the useful way to picture it. With two inputs, every example is a point on a flat plane, at position (x₁, x₂).

The boundary where the sum exactly equals the threshold, w₁x₁ + w₂x₂ = θ, is a straight line. Points on one side get a 1. Points on the other side get a 0.

So a perceptron is a line-drawer. Its weights decide where the line sits and which way it tilts.

With two inputs, the perceptron's rule splits the plane with a straight line: one side outputs 1, the other outputs 0.

Learning from mistakes

A new perceptron starts with weights that put its line in the wrong place. So how does it get better?

1. It looks at one example at a time and makes a guess.
2. If the guess is right, nothing changes.
3. If the guess is wrong, it nudges each weight by the input times the error.

As a formula: w ← w + x · error, where the error is the target minus the output. In practice, the nudge is often scaled down by a small number called the learning rate, and the threshold is adjusted the same way.

A worked example

Say the perceptron outputs 0 for a point, but the right answer was 1.

• The error is 1 − 0 = +1.
• So each weight moves by its own input. Weights for strong inputs move the most.
• The line swings toward that point, until the point ends up on the correct side.

If it had output 1 when the answer was 0, the error would be −1, and the weights would move the other way.

Target 1, output 0: the error is +1, so the weights grow and the line swings toward the missed point.

When it works, and when it doesn't

Repeat this, example after example, and the line keeps moving until every point lands on the right side.

Rosenblatt proved something remarkable: if one straight line can separate the two classes, this rule always finds such a line, in a finite number of steps. This is known as the perceptron convergence theorem.

The catch is in that "if." Some problems have no single separating line at all. The classic example is XOR, and solving it takes more than one neuron.

Example after example, the line settles where it separates every point. This is guaranteed whenever a straight line can do it.

Quick check

The perceptron gets a point right. Do its weights change, or stay the same?

Answer: they stay the same. When the output is correct, the error is zero, and input times zero changes nothing. A perceptron only learns from its mistakes.

Key takeaways

• A perceptron is one neuron: a weighted sum, then a threshold that outputs 1 or 0.
• With two inputs, its decision boundary is a straight line.
• It learns by nudging each weight by the input times the error, and only when it's wrong.
• If a straight line can separate the classes, it is guaranteed to find one.

This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.