More on the need for non-linear activation functions We mentioned in Section 7.2 that one of the reasons we use non-linear activation functions for each layer in a neural network is that if we did not, the resulting network is exactly equivalent to a single-layer network. Now that we have the notation for multilayer networks, we can see that intuition is more detail. Imagine the first two layers of such a network of purely linear layers: