# Without one bend, 100 layers act like one

Deep networks stack layer after layer. But there's a catch hiding in that stack. Without one small ingredient, a hundred layers can do exactly what one layer does, and nothing more.

That ingredient is the activation function. Here's why it matters, in a few lines of algebra.

## What a layer does

Every layer in a neural network does two things:

1. It multiplies and adds: each input is multiplied by a weight, the results are summed, and a bias is added.
2. It passes the result through an activation function: a small bend.

The first step is simple arithmetic. The second step is the one that's easy to overlook, and it's the one that makes depth worth having.

## What happens without the bend

Let's remove the activation function and watch what happens with two tiny layers.

- Layer one doubles its input and adds one: `h = 2x + 1`.
- Layer two triples that and subtracts four: `y = 3h − 4`.

Now put them together. Substitute the first into the second: `y = 3(2x + 1) − 4 = 6x − 1`.

Two layers, and the result is a single multiply-and-add. It's still just one straight line.

![Two layers without a bend collapse into a single line, 6x − 1](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/006fuly20gs5m00ukypl.png)
*Two layers without a bend: 2x + 1, then 3h − 4. Combined, they collapse into a single line, 6x − 1.*

## Stacking doesn't help

This isn't a coincidence of the numbers we picked. Any multiply-and-add fed into another multiply-and-add is still a multiply-and-add.

So stack a hundred of these layers, and they still collapse into one. All that depth, and the network can only draw a straight line.

## Add the bend: ReLU

Now add an activation function. The most popular one is called ReLU (rectified linear unit). Its rule is:

- Negative numbers become zero.
- Positive numbers pass straight through.

So −3 becomes 0, and 2 stays 2. That's it. It looks almost too simple to matter.

![ReLU: negative inputs become zero, positive inputs pass straight through](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/t1hzjnnkmlxengv6oh9h.png)
*ReLU: negative inputs become zero, positive inputs pass straight through.*

## From bends to curves

That one kink at zero is enough to stop the collapse. The layers can't be merged into a single line anymore.

Better still, bends can be combined. Add two ReLUs, one of them flipped, and you get a peak. Combine many of them, with different positions and slopes, and they trace a curve, one straight piece at a time.

This is how networks build complicated functions out of very simple parts.

![Many ReLU pieces combine to trace a curve](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/vkr6crgjzzjm6iyvuhga.png)
*Many ReLU pieces, each a straight segment with one bend, combine to trace a curve.*

## Sigmoid vs ReLU

Older networks often used a smooth S-shaped curve called sigmoid, which squashes any number into the range between 0 and 1.

Today, ReLU is the standard choice in most networks, because it's simple and fast to compute. It also tends to make deep networks easier to train.

![Sigmoid, a smooth S-curve, compared with ReLU](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/z3hhdvvtjk7gg08o4xa5.png)
*Sigmoid, a smooth S-curve, was common in older networks. ReLU is the standard choice today.*

## Quick check

A network has ten layers, with no activation functions. Can it draw a curve, or only a straight line?

**Answer: only a straight line.** Without a bend, all ten layers collapse into one multiply-and-add.

## Key takeaways

- Every layer multiplies, adds, and then applies an activation function.
- Without the activation, any stack of layers collapses into a single straight line, like 2x + 1 and 3h − 4 becoming 6x − 1.
- ReLU turns negatives into zero and passes positives through. That one bend stops the collapse.
- Many bends combined can trace curves.
- Sigmoid was common in older networks. ReLU is the default today.

---

*This article is part of Deep Learning, Visually by Logit Solutions AI: short animated explainers that cover one idea at a time, each with a written version like this one.*