# Too big, it explodes. Too small, it crawls.

Backpropagation tells every weight which way to move: its gradient. It doesn't say how far. That's the job of the learning rate, written η (eta). Every step of gradient descent is

`w ← w − η ∇L`

The gradient comes from the model. The learning rate is a setting you choose, and it is the first one people tune. This article shows what it does on the smallest possible example, why there is a hard limit on how big it can be, and how practitioners pick it and change it during training. The ideas follow the CS878 Deep Learning lecture on learning rates.

## One weight on a bowl

Take a single weight and the loss L(w) = w², a bowl with its bottom at w = 0. The gradient is 2w, so one step is

`w ← w − η · 2w = (1 − 2η) · w`

Start at w = 2, where the loss is 4, and try four learning rates:

- **η = 0.01:** each step multiplies w by 0.98. After ten steps, w has only reached 1.634 and the loss is still 2.67. It crawls.
- **η = 0.4:** each step multiplies w by 0.2. Two becomes 0.4, then 0.08, then 0.016. Fast.
- **η = 0.9:** each step multiplies w by −0.8. The weight jumps to the other side of the bowl each time (−1.6, 1.28, −1.024, 0.819), but the jumps shrink, so it still lands.
- **η = 1.1:** each step multiplies w by −1.2. The jumps grow: −2.4, 2.88, −3.456, 4.147. The loss climbs 5.76, 8.29, 11.94, 17.20. It explodes.

![One weight on the bowl L = w², four learning rates: crawls, fast, zig-zags, explodes.](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/70kyq7bw435awadh3if1.png)

*One weight on the bowl L = w², four learning rates: crawls, fast, zig-zags, explodes.*

## Why the edge is at one

Everything depends on the factor 1 − 2η. After t steps, w_t = (1 − 2η)^t · w₀.

- Below η = 0.5, the factor is between 0 and 1: the weight slides in without overshooting.
- At η = 0.5, the factor is 0: one jump lands exactly on the bottom.
- Between 0.5 and 1, the factor is between −1 and 0: the weight flips sides but shrinks.
- At η = 1, the factor is −1: it bounces between 2 and −2 forever.
- Past 1, the factor is below −1: it grows every step.

Nothing here is special to w². For any bowl L = (λ/2) w², with curvature λ (the second derivative), the factor is 1 − ηλ. So training is stable only if 0 < η < 2/λ, and one jump lands at η = 1/λ. For L = w², λ = 2: the edge is 1 and the one jump is 0.5.

![The factor 1 − 2η decides everything: slides in below 0.5, one jump at 0.5, zig-zags up to 1, explodes past 1.](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/qmp5namabvxoij2ljcyn.png)

*The factor 1 − 2η decides everything: slides in below 0.5, one jump at 0.5, zig-zags up to 1, explodes past 1.*

## Millions of directions

A real network has millions of weights, so the loss has millions of directions, and they are not equally steep. Take two weights and L = ½(w₁² + 10 w₂²): one direction with curvature 1 (flat), one with curvature 10 (steep). Each step multiplies w₁ by (1 − η) and w₂ by (1 − 10η).

Starting from (4, 1):

- **η = 0.02:** the steep direction shrinks fast (×0.8 per step) but the flat one barely moves (×0.98). After ten steps the weights are at (3.27, 0.11): done on the steep side, stuck on the flat side.
- **η = 0.21:** 0.21 is past 2/10, the edge of the steep direction. w₂ is multiplied by −1.1 each step: 1, −1.1, 1.21, −1.331, 1.464. It explodes.
- **η = 2/11 ≈ 0.18:** the best single fixed rate, 2/(λ_min + λ_max). Both directions shrink by 9/11 per step, and after ten steps the weights are at (0.54, 0.13).

The steep direction sets the speed limit, η < 2/λ_max, and the flat direction then crawls. Even the best fixed rate shrinks the error by only (κ − 1)/(κ + 1) = 9/11 per step, where κ = λ_max/λ_min = 10 is how uneven the valley is.

![The valley L = ½(w₁² + 10w₂²): the steep side sets the speed limit; the best fixed step is 2/11.](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/6ertqowbcy3h6syw9kgt.png)

*The valley L = ½(w₁² + 10w₂²): the steep side sets the speed limit; the best fixed step is 2/11.*

## Picking a starting rate

Two quick checks get you most of the way:

1. Before training, look at the first loss. For softmax with C classes and small random weights, it should be about ln C, which is 2.30 for ten classes. If it isn't, something else is wrong.
2. Try a few learning rates a factor of ten apart: 1e-1, 1e-2, 1e-3, 1e-4. Run each for about a hundred iterations and keep the one whose loss drops fast without blowing up. Too small barely moves; too big turns into NaN.

![Trying 1e-4 to 1e-1 from the same starting loss of 2.30: keep the one that drops fast.](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/w1vluvc1ox9dy2xthu6u.png)

*Trying 1e-4 to 1e-1 from the same starting loss of 2.30: keep the one that drops fast.*

## Try it in PyTorch

The bowl example, at the learning rate that explodes:

```python
w, lr = 2.0, 1.1
for t in range(4):
    w = w - lr * 2 * w
    print(round(w, 3))
```

It prints −2.4, 2.88, −3.456, 4.147. Change `lr` to 0.5 and w lands on 0 in one step.

In a real training loop the learning rate is one argument of the optimizer. A quick sweep tries a few values a factor of ten apart:

```python
import torch
# model = your network
for lr in (1e-1, 1e-2, 1e-3, 1e-4):
    opt = torch.optim.SGD(model.parameters(), lr=lr)
    ...            # train ~100 iterations, keep the lr whose loss drops fast
```

Shrinking the rate during training (decay schedules) is its own episode.

## Key takeaways

- The learning rate sets how far each gradient step goes; it is chosen, not learned.
- On a bowl with curvature λ, each step multiplies the error by 1 − ηλ: stable only if η < 2/λ, one jump at η = 1/λ.
- In a real network the steepest direction sets the speed limit, so the flat directions crawl.
- Start by checking the first loss (ln C), then try rates a factor of ten apart and keep the one that drops fast.

---

*This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.*