Explanation of Neural Network From Maximum Likelihood Estimation

Sigmoid

2026, June, 08

From a statistical learning perspective, modern neural networks can indeed be understood as a large-scale maximum likelihood estimation (MLE) process. Specifically, a neural network is a parameterized function, and the most common way to train a neural network is to perform maximum likelihood estimation on the data.

Coin Tossing

MLE (Maximum Likelihood Estimation) is one of the core ideas in statistics.

One-sentence summary:

MLE selects, from all possible model parameters, the one that "makes the observed data most likely."

Below, you find a damaged coin with uneven texture. So you don't know if it is fair. That is, when you toss this coin, you don't know the probability of heads or tails. You want to know: what is the probability $p$ of getting heads.

Thus, you decide to toss it 10 times consecutively and obtain the following result: heads 8 times, tails 2 times.

Suppose this were a fair coin. Then we set the prior hypothesis $p=0.5$ .

Under this prior hypothesis, the probability of obtaining the above result is:

P(D|p=0.5) = 0.5^{10} \approx 0.000976

That is, if this were a fair coin, then the probability of getting 8 heads and 2 tails in 10 tosses is about $0.000976$ — clearly, this number is considered impossible in probability and statistics. So you can clearly feel that this coin is not fair.

Now we propose another hypothesis: suppose the probability of heads is $p=0.8$ . Then the probability of the above result is:

P(D|p=0.8) = 0.8^8 \times 0.2^2 \approx 0.0067

We also find that when $p=0.8$ , the probability of the above result is maximized, achieving the maximum value $0.0067$ .

0.0067 > 0.000976

This shows that if the probability of heads is 0.8, then the likelihood of observing this data is greatest.

Thus: $\hat p=0.8$

This is maximum likelihood estimation.

Likelihood Function

Given data:

D = \{x_1, x_2, \cdots, x_n\}

Suppose we have model parameters: $\theta$ .

The likelihood function is defined as:

L(\theta) = P(D|\theta)

Maximum Likelihood Estimation (MLE) is:

\hat\theta = \arg\max_\theta P(D|\theta) = \arg\max_\theta L(\theta)

Many people feel confused when first learning this:
Since $P(D|\theta)$ and $L(\theta)$ have exactly the same expression, why change the name?

The key lies in different perspectives:

Probability $P(D|\theta)$ perspective:
The parameter $\theta$ is a fixed known value, while the data $D$ is random.
We ask: "Given the parameter, how likely are different data outcomes?"
Likelihood $L(\theta)$ perspective:
The data $D$ is a fixed observed value, while the parameter $\theta$ is a varying unknown quantity.
We ask: "Under different parameter values, how likely is this observed data?"

According to the conditional formula:

P(D|\theta) = \prod_{x_i \in D} P(x_i|\theta)

If the data $D$ contains many samples, multiplying many small numbers yields a result approaching zero, which can cause numerical underflow in computers.

Thus we take the logarithm, converting multiplication into addition:

\ell(\theta|D)= \log P(D|\theta) = \sum_{x_i \in D} \log P(x_i|\theta)

After taking the logarithm, the values stay within an acceptable range, making optimization easier.

Therefore, almost all practical training optimizes:

\max_\theta \ell(\theta|D)

This form is called Log-Likelihood.

In the likelihood function, we need to find the parameter $\hat{\theta}_\text{MLE}$ that maximizes the likelihood.

Solving for Parameters

If $\ell(\theta|D)$ is differentiable, $\hat{\theta}_{\rm MLE}$ typically satisfies:

\frac{\partial \ell(\theta|D)}{\partial \theta} = 0

And the second-order condition ensures a maximum:

\frac{\partial^2 \ell(\theta|D)}{\partial \theta^2} < 0

Take the simplest Bernoulli distribution as an example:

Data $x_i \in \{0,1\}$
Model: $P(X=1)=p, P(X=0)=1-p$

We have the likelihood function:

L(p|D) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i}

Convert to log-likelihood:

\ell(p|D) = \sum_{i=1}^n x_i \log p + (1-x_i) \log (1-p)

Take the derivative with respect to $p$ and set to 0:

\frac{\partial \ell}{\partial p} = \frac{\sum x_i}{p} - \frac{n-\sum x_i}{1-p} = 0

Solve to obtain:

\hat{p}_{\rm MLE} = \frac{\sum_{i=1}^n x_i}{n}

In neural networks, the MLE parameter $\theta$ corresponds to the network's weights and biases, typically denoted as:

\theta = {W_1, b_1, W_2, b_2, \dots, W_L, b_L}

where $L$ is the number of layers, $W_i$ is the weight matrix of the $i$ -th layer, and $b_i$ is the bias vector.

In other words: a neural network is a function family $f_\theta(x)$

It defines the conditional probability distribution:

P_\theta(y|x)

Training a neural network = selecting parameters $\theta$ that maximize the probability of the observed data:

\hat{\theta}_{\rm MLE} = \arg\max_\theta \prod_i P_\theta(y_i | x_i)

Take classification as an example: a binary classification neural network outputs:

\hat{y} = \sigma(f_\theta(x))

We see that maximizing the likelihood can be written as minimizing the cross-entropy loss function as the cost function:

\hat{\theta}_{\rm MLE} = \arg\max_\theta \prod_i \hat{y}_i^{y_i} (1-\hat{y}_i)^{1-y_i}

Finally, we obtain the cross-entropy loss:

L(\theta) = - \sum_i \Big[ y_i \log \hat{y}_i + (1-y_i) \log (1-\hat{y}_i) \Big]

Here $\theta$ represents all weights and biases.

Take regression as another example: suppose

y_i = f_\theta(x_i) + \epsilon_i, \quad \epsilon_i \sim N(0, \sigma^2)

Log-likelihood:

\ell(\theta) = -\frac{1}{2\sigma^2} \sum_i (y_i - f_\theta(x_i))^2 + C

We see that the maximum likelihood objective is equivalent to minimizing the mean squared error, where $\theta$ = the weights and biases of the neural network.

The hidden assumption here is that the errors follow a Gaussian distribution. If we instead assume a Laplace distribution, MLE would lead to the MAE loss. This explains why different tasks choose different loss functions — they correspond to different assumptions about the data distribution.

In neural networks, maximum likelihood estimation means finding, among all possible combinations of weights and biases, the set that makes the training data most likely to occur.

Therefore, every time we use gradient descent to update weights, we are essentially performing MLE optimization.