Nonlinearity with Activations

Linear Combinations

The concept of linearity runs through mathematics, physics, engineering, and many other fields. Simply put, linearity describes two fundamental operational properties: additivity and homogeneity.

A relationship, function, or system is said to be linear if it satisfies the following two conditions simultaneously.

Consider a transformation or function $f$ acting on inputs $x$ and $y$ .

Additivity

The sum of parts equals the whole. That is, processing two inputs separately and then adding the results is the same as adding the inputs first and then processing the sum.

f(x + y) = f(x) + f(y)

Example: Multiplication $f(x) = 5x$ satisfies additivity.
Left: $f(2+3) = 5 \times (2+3) = 25$ , Right: $f(2) + f(3) = 10 + 15 = 25$ , equal.
Counterexample: The square function $f(x) = x^2$ does not satisfy additivity.
Left: $(2+3)^2 = 25$ , Right: $2^2 + 3^2 = 4+9=13$ , not equal.

Homogeneity

Scaling the input scales the output by the same factor. That is, multiplying an input by a constant (scalar) yields the original output multiplied by the same constant.

f(c \cdot x) = c \cdot f(x) \quad (\text{c is any constant})

Example: $f(x) = 5x$ satisfies homogeneity.
Left: $f(4 \times 3) = 5 \times 12 = 60$ , Right: $4 \times f(3) = 4 \times 15 = 60$ , equal.
Counterexample: Adding a constant $f(x) = 5x + 2$ does not satisfy homogeneity.
Left: $f(4 \times 3) = 5 \times 12 + 2 = 62$ , Right: $4 \times f(3) = 4 \times (15+2) = 68$ , not equal.

In the definition above, satisfying both additivity and homogeneity is sufficient and necessary for linearity.

Many people mistakenly think that "linear" simply means "the graph is a straight line." This understanding is inaccurate.

Truly linear: A straight line that must pass through the origin. For example, $y = 2x$ . It satisfies both additivity and homogeneity. If the input is 0, the output must also be 0.

Strictly speaking, if a bias term $b$ is added, it becomes affine. It is a straight line, but not necessarily passing through the origin. For example, $y = 2x + 1$ . It does not satisfy homogeneity (since $f(0)=1 \neq 0$ ). Although it is often called a "linear function," in the strict definition it is not linear, but "linear + translation," also known as affine.

Now we can understand why it's called a "linear" combination. This combination allows only two types of "linear" operations:

Scalar multiplication (corresponding to homogeneity): Stretching or compressing a vector by a factor $c$ .
Addition (corresponding to additivity): Adding the processed vectors together.

Nonlinear operations like squaring, taking absolute values, or logarithms are not allowed. Therefore, expressions like $c_1 v_1^2 + c_2 v_2$ or $c_1 v_1 + c_2 v_2 + 3$ are not called linear combinations.

It can be said that "linear" is the simplest and most core relationship in mathematics and science because it allows us to decompose complex problems into a superposition of simple parts.

Affine

An affine function $f: \mathbb{R}^n \to \mathbb{R}^m$ is defined as:

f(x) = Ax + b

where $A$ is a matrix and $b$ is a constant vector. It is a linear function plus a bias.

If you take a weighted linear combination (coefficients can be scalars) of affine functions:

g(x) = \sum_{i=1}^k \alpha_i f_i(x)

where each $f_i(x) = A_i x + b_i$ .

Expanding:

g(x) = \sum_{i=1}^k \alpha_i (A_i x + b_i) = \left(\sum_{i=1}^k \alpha_i A_i \right) x + \sum_{i=1}^k \alpha_i b_i

This is still an affine function. Thus, weighted combinations do not destroy affinity.

If you combine affine functions using nonlinear polynomial operations, such as squaring or higher powers:

h(x) = (f_1(x))^2 + 3 f_2(x) f_3(x) + 5

Note: $f_1(x)^2$ is element-wise squaring, $f_2(x) f_3(x)$ is element-wise multiplication or dot product.

These operations introduce quadratic or higher-order terms and no longer maintain the $Ax+b$ form. Therefore, nonlinear polynomial combinations are generally not affine functions.

Affine Composition

This question is very important and key to understanding the nature of neural networks. Let's carefully analyze whether the composition of affine functions remains affine.

Suppose you have two affine functions:

f(x) = A x + b, \quad g(x) = C x + d

Composition is:

h(x) = f(g(x)) = f(Cx + d)

Expanding to see the form:

h(x) = A(Cx + d) + b = (AC)x + (Ad + b)

The result is still an affine function.

Thus, the composition of affine functions remains affine. This can be generalized: composing any number of affine functions:

f_n(\dots f_2(f_1(x))\dots)

can always be written as:

A_\text{total} x + b_\text{total}

This is also why: if each layer of a neural network is affine without activation functions, no matter how many layers are stacked, the network's expressive power is the same as a single layer. We can think of an affine layer as an operation of stretching, rotating, or translating, or a combination of these. Composing multiple affine transformations is equivalent to continuous stretching + rotation + translation → ultimately, a single overall stretching + rotation + translation. Therefore, composing affine transformations does not increase nonlinear degrees of freedom and cannot represent curves or complex functions.

Each layer of a neural network is an affine mapping:

f(x) = Wx + b

If $W$ is invertible (i.e., $\det(W) \neq 0$ ), then $f$ is bijective and continuous.
The inverse mapping is:

f^{-1}(y) = A^{-1}(y - b)

which is also affine and continuous → a homeomorphism. However, there is a core condition: $W$ must be invertible.

Composing multiple affine transformations is equivalent to composing multiple homeomorphisms. Suppose you have multiple consecutive invertible affine maps:

f_n \circ f_{n-1} \circ \cdots \circ f_1

Each $f_i$ is a homeomorphism, and the composition of homeomorphisms remains a homeomorphism. Hence, the final mapping is also a homeomorphism.

Thus, we have the following conclusion: Let $M$ be the original manifold:

An affine mapping that is continuous and bijective → the manifold structure is preserved (homeomorphic).
Composing multiple such affine maps continuously → the resulting manifold is homeomorphic to $M$ .

In other words: "Continuous linear + bias transformations do not change the topology of the manifold; they only stretch/rotate/translate/shear it."

This is why using only linear layers cannot "bend" the data manifold: the data retains its "original shape" (topologically invariant) under nested linear layers. If a decision boundary needs to cut a complex manifold, nonlinearity (activation functions) must be introduced.

This is also the goal in deep learning. If the original manifold's structure is inseparable for a given task, but separability is required for a learning objective, then the original manifold and the target manifold must not be homeomorphic.

In the transformation from the original manifold to the target manifold, losing homeomorphism requires breaking away from linear, affine transformations. Because affine transformations (or more generally, homeomorphic transformations) alone are usually insufficient for disentangling classes. Deep networks often need to reconstruct the data manifold through non-homeomorphic nonlinear operations to make it ultimately linearly separable.

We need to consider how to make the composition of affine transformations nonlinear. This insight is the core of the geometric interpretation of neural networks:

Linear layers can only perform topology-preserving affine transformations; nonlinear layers allow topological changes, enabling neural networks to represent complex functions.

Activation for Affine Layers

Each layer of a neural network is an affine mapping:

l_i(x) = W_i x + b_i

where $i$ denotes the $i$ -th layer. The nested composition of affine mappings across $n$ layers is:

l_n \circ l_{n-1} \circ l_{n-2} \circ \dots \circ l_0

If no activation function is introduced, the result above is still an affine function.

Proof is simple:

l_2(l_1(x)) = W_2(W_1x+b_1)+b_2 = (W_2W_1)x+(W_2b_1+b_2)

which remains in the form $Ax+b$ .

By induction:

l_n \circ \cdots \circ l_0 = Wx + b

Therefore, network depth loses its meaning entirely.

To introduce nonlinearity, we can add activation functions to each layer:

f(x) = (\sigma_n \circ l_n) \circ (\sigma_{n-1} \circ l_{n-1}) \circ \cdots \circ (\sigma_0 \circ l_0)(x)

where

l_i(x)=W_i x+b_i

Expanded, this is:

f(x) = \sigma_n \Bigl( W_n \sigma_{n-1} \bigl( W_{n-1} \cdots \sigma_0(W_0x+b_0) \cdots +b_{n-1} \bigr) + b_n \Bigr)

This is the standard form of a neural network.

With activation functions added:

(\sigma \circ l_2) \circ (\sigma \circ l_1)

cannot be simplified to

Wx+b

because generally:

\sigma(Ax+b) \neq A'\sigma(x)+b'

The activation function breaks the closure property of affine mappings under composition. In other words, it breaks the two key characteristics of additivity and homogeneity. This is essentially the source of the nonlinear expressive power of neural networks.

From the manifold perspective discussed earlier, this can be further written as:

M \xrightarrow{l_0} M_1 \xrightarrow{\sigma_0} M_1' \xrightarrow{l_1} M_2 \xrightarrow{\sigma_1} M_2' \rightarrow \cdots

where: affine layers are responsible for stretching, rotating, translating, shearing; activation layers are responsible for folding, cropping, compressing, and restructuring the manifold.

If all $W_i$ are invertible, then the affine layers themselves are only homeomorphic transformations and do not change the topology of the manifold. What gradually makes the manifold separable is often precisely the nonlinear deformation introduced by activation functions.

Therefore, from a geometric perspective, a neural network can be understood as an alternating combination of affine transformations and nonlinear manifold operations acting on the data manifold. This perspective aligns with many interpretations of modern manifold learning and geometric deep learning.

Activations

There is no "single best" choice for an activation function. However, from the perspective of trainability and expressive power of neural networks, a good activation function generally should satisfy several core characteristics:

Nonlinearity

As discussed earlier, nonlinearity is needed to break the linear closure of stacked affine layers, allowing multi-layer networks to approximate arbitrary functions.

Mathematical requirement: It must not satisfy linear combination closure:

\sigma(a x + b y) \neq a \sigma(x) + b \sigma(y)

Examples: ReLU, Tanh, Sigmoid, GELU.

Linear functions are not suitable because the composition of multiple linear layers remains linear, which would degenerate back to affine composition and limit expressive power.

Differentiability

Because the core of neural networks relies on backpropagation to determine weights, the activation function must be differentiable over most of its domain to facilitate gradient calculation during backpropagation. For non-differentiable points (e.g., ReLU at 0), subgradients are typically used. Moreover, the derivative properties affect gradient flow: gradients that are too small can cause vanishing gradients (Sigmoid), while gradients that are too large can cause exploding gradients.

Continuity

Ensures that the network output changes continuously with the input; otherwise, training becomes unstable. Having few and controllable discontinuities is usually acceptable (e.g., ReLU has only one point of discontinuity, which has little practical impact).

Among the properties above, meeting the conditions is merely an "entry ticket," not a "guarantee of success."