最大似然估计

数据$X$
$$
X = \begin{pmatrix}
{x_{1}}&{x_{2}}&{\cdots}&{x_{N}}
\end{pmatrix}^T
= \begin{pmatrix}
{x_{11}}&{x_{12}}&{\cdots}&{x_{1p}}\\
{x_{21}}&{x_{22}}&{\cdots}&{x_{2p}}\\
{\vdots}&{\vdots}&{\ddots}&{\vdots}\\
{x_{N1}}&{x_{N2}}&{\cdots}&{x_{Np}}\\
\end{pmatrix}$$

$$x_{i} \in \mathbb{R}^{p}$$
$$x_{i} \sim N(\mu, \Sigma)$$

$$\theta=(\mu, \Sigma)$$

最大似然估计MLE: $\theta_{MLE}=\arg \max _{\theta} p(X | \theta)$
为了简化计算，下面我们推导一元高斯分布的最大似然估计。令 p=1, $ \theta=\left(\mu, \sigma^{2}\right)$
$$p(x)=\frac{1}{\sqrt{2 \pi} \sigma} \exp \left(-\frac{(x-\mu)^{2}}{2 \sigma^{2}}\right)$$

\begin{aligned}
\log P(X | \theta) &=\log \prod_{i=1}^{N} p\left(x_{i} | \theta\right)=\sum_{i=1}^{N} \log p\left(x_{i} | \theta\right) \\
&=\sum_{i=1}^{N} \log \frac{1}{\sqrt{2 x} \sigma} \exp \left(-\frac{\left(x_{i} - \mu\right)^{2}}{2 \sigma^{2}}\right) \\
&=\sum_{i=1}^{N}\left[\log \frac{1}{\sqrt{2\pi}}+{\log \frac{1}{\sigma}} -\frac{\left(x_{i}-\mu\right)^{2}}{2 \sigma^{2}}\right] \\
\end{aligned}

\begin{aligned}
\mu_{MLE} &=\arg\max_{\mu} \log P(X|\theta) \\
&= \arg\max_{\mu} \sum_{i=1}^{N} - \frac{(x_{i}-\mu)^{2}}{2 \sigma^{2}} \\
&= \arg\max_{\mu} \sum_{i=1}^{N}(x_{i}-\mu)^{2} \\
\end{aligned}

对$\mu$求导，并令其为$0$，
$$\frac{\partial}{\partial \mu} \Sigma\left(x_{i}-\mu\right)^{2}=\sum_{i=1}^{N} 2 \cdot\left(x_{i}-\mu\right) \cdot(-1)=0$$
$$\sum_{i=1}^{N}\left(x_{i}-\mu\right)=0$$
$$\mu_{MLE} = \frac{1}{N}\sum_i^N x_i$$

下面证明最大似然估计的均值无偏：
$$E[\mu_{MLE}] = E[\frac{1}{N}\sum_i^N x_i]=\mu$$

\begin{aligned}
\sigma^2_{MLE} &= \arg\max_{\sigma} \log P(X|\theta) \\
&=\arg\max_{\sigma} \sum_{i=1}^{N} \left(-\log \sigma -\frac{1}{2\sigma^2} \left(x_{i} - \mu\right)^{2}\right)\\
&= \arg\max_{\sigma} \mathcal{L}(\sigma) \\
\end{aligned}

对上式$\sigma$求导并令其为$0$，

$$\frac{\partial \mathcal{L}} {\partial \sigma} = \sum_{i=1}^{N}\left[-\frac{1}{\sigma}+\frac{1}{2}\left(x_{i}-\mu\right)^{2} \cdot(+2) \sigma^{-3}\right]^{2} =0 $$

$$\sigma^2_{MLE} = \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2$$

方差有偏:
\begin{aligned}
E\left[\sigma_{MLE}^{2}\right]=\frac{1}{N} E\left[\sum_{i=1}^{N}\left(x_{i}-\mu_{MLE}\right)^{2}\right] &= E\left[\frac{1}{N} \sum_{i=1}^N x_{i}^{2} -2\frac{1}{N}\sum_{i=1}^N x_i \mu_{MLE} +\mu_{MLE}^{2} \right] \\
&= E\left[\frac{1}{N} \sum_{i=1}^N x_{i}^{2} -\mu_{MLE}^{2} \right] \\
\end{aligned}

\begin{aligned}
E[\sigma_{MLE}^2] &=E\left[\frac{1}{N}\sum_{i=1}^{N} x_{i}^{2}-\mu^{2}+\mu^{2}-\mu_{MLE}^{2}\right] \\
&=E\left[\frac{1}{N} \sum_{i=1}^{N} x_{i}^{2}-\mu^{2}\right]+E\left[\mu^{2}-\mu_{MLE}^{2}\right] \\
&=E\left[\frac{1}{N} \sum_{i=1}^{N}\left(x_{i}^{2}-\mu^{2}\right)\right]-E\left[\mu_{MLE}^{2} - \mu^{2}\right] \\
&= \sigma^2 - (E[\mu_{MLE}^2] - E[\mu^2]) \\
&= \sigma^2 - (E[\mu_{MLE}^2] - E^2 [\mu_{MLE}]) \\
&= \sigma^2 - Var[\mu_{MLE}] \\
&= \frac{(N-1)\sigma^2}{N}
\end{aligned}

\begin{aligned}
Var[ \mu_{M L E}] &=\operatorname{Var}\left[\frac{1}{N} \sum_{i=1}^{N} x_{i}\right] \\
&= \frac{1}{N^{2}} \sum_{i=1}^{N} Var\left[x_{i}\right] \\
&=\frac{\sigma^{2}}{N}
\end{aligned}