$$\begin{aligned}
&D=\left\lbrace(x_1, y_1),\left(x_{2}, y_{2}\right), \cdots,\left(x_{N}, y_{N}\right)\right\rbrace \\
&x_{i} \in \mathbb{R}^{p}, \quad y_{i} \in \mathbb{R}, \quad i=1,2, \cdots, N
\end{aligned}$$
$$
X = \begin{pmatrix}
{x_{1}}&{x_{2}}&{\cdots}&{x_{N}}
\end{pmatrix}^T
= \begin{pmatrix}
{x_{11}}&{x_{12}}&{\cdots}&{x_{1p}}\\
{x_{21}}&{x_{22}}&{\cdots}&{x_{2p}}\\
{\vdots}&{\vdots}&{\ddots}&{\vdots}\\
{x_{N1}}&{x_{N2}}&{\cdots}&{x_{Np}}\\
\end{pmatrix}$$
$$
Y=\left(\begin{array}{c}
{y_{1}} \\
{y_{2}} \\
{\vdots} \\
{y_{N}}
\end{array}\right)_{N \times 1}
$$
岭回归
频率角度
$$
\begin{aligned} J(w) &=\sum_{i=1}^{N}\left|\left| w^{\top} x_{i}-y_{i}\right|\right|^{2}+\lambda w^{\top} w \\
&=\left(w^{\top} X^{\top}-Y^{\top}\right)(X w-Y)+\lambda w^{\top} w \\
&={w^{\top} \left(X^{\top} X +\lambda I \right)w} -2 w^{\top} X^{\top} Y+Y^{\top} Y
\end{aligned}
$$
$$\hat{w}= \arg \min_w J(w)$$
\begin{aligned}
\frac{\partial J(w)}{\partial w}=2\left(X^{\top} X+\lambda I\right) w &-2 X^{\top} Y=0 \\
\hat{w}=\left(X^{\top} X+\lambda I\right)^{-1} X^{\top} Y
\end{aligned}
贝叶斯角度
$$f(w)=w^{\top} x_i$$ $$y=f(w)+\varepsilon=w^{\top} x_i+\varepsilon$$ $$\varepsilon \sim N\left(0, \sigma^{2}\right)$$ $$y_i | x_i;w = N\left(w^{\top} x_i, \sigma^2 \right)$$
$$p(y_i | w)=\frac{1}{\sqrt{2\pi} \sigma} \exp \left\lbrace-\frac{\left(y_i-w^{\top} x_i\right)^{2}}{2\sigma^{2}}\right\rbrace$$ $$p(w)=\frac{1}{\sqrt{2 \pi} \sigma_{0}^{2}} \exp \lbrace-\frac{|| w ||^{2}}{2 \sigma_{0}^{2}}\rbrace$$ $$p(y_i | w) \cdot p(w)=\frac{1}{\sqrt{2\pi}\sigma} \cdot \frac{1}{\sqrt{2\pi} \sigma_0} \cdot \exp \left\lbrace-\frac{\left(y_i-w^{\top} x_i\right)^{2}}{2 \sigma^{2}}-\frac{||w||^{2}}{2 \sigma_{0}^{2}}\right\rbrace$$
最大后验概率估计MAP：
\begin{aligned}
\hat{w} &=\arg\max_{w} p(w | Y) \\
&=\arg\max_{w}\prod_{i=1}^N p(y_i | w) \cdot p(w) \\
&=\arg \max_w \sum_{i=1}^N \log [p(y_i | w) \cdot p(w)] \\
&= \arg \max_w \sum_{i=1}^N -\frac{\left(y_i-w^{\top} x_i\right)^{2}}{2 \sigma^{2}}-\frac{||w||^{2}}{2 \sigma_{0}^{2}} \\
&= \arg \min_w\sum_{i=1}^N {\left(y_i-w^{\top} x\right)^{2}}+\frac{N\sigma^2 }{ \sigma_{0}^{2}} ||w||^{2}
\end{aligned}
总结
最小二乘估计等价于噪声为高斯分布的最大似然估计，二范数为正则项的最小二乘估计等价为先验为高斯分布的最大后验估计。
