$$\begin{aligned}
&D=\left\lbrace(x_1, y_1),\left(x_{2}, y_{2}\right), \cdots,\left(x_{N}, y_{N}\right)\right\rbrace \\
&x_{i} \in \mathbb{R}^{p}, \quad y_{i} \in \mathbb{R}, \quad i=1,2, \cdots, N
\end{aligned}$$
$$
X = \begin{pmatrix}
{x_{1}}&{x_{2}}&{\cdots}&{x_{N}}
\end{pmatrix}^T
= \begin{pmatrix}
{x_{11}}&{x_{12}}&{\cdots}&{x_{1p}}\\
{x_{21}}&{x_{22}}&{\cdots}&{x_{2p}}\\
{\vdots}&{\vdots}&{\ddots}&{\vdots}\\
{x_{N1}}&{x_{N2}}&{\cdots}&{x_{Np}}\\
\end{pmatrix}$$
$$
Y=\left(\begin{array}{c}
{y_{1}} \\
{y_{2}} \\
{\vdots} \\
{y_{N}}
\end{array}\right)_{N \times 1}
$$
最小二乘估计：
$$
\begin{aligned}
L(w) &=\left(\sum_{i=1}^{N}\left|\left|w^{\top} x_{i}-y_{i}\right|\right|^{2}\right) \\
&=\sum_{i=1}^{N}\left(w^{\top} x_{i}-y_{i}\right)^{2} \\
&=\left( w^{\top} x_{1}-y_{1} \quad w^{\top} x_{2}-y_{2} \quad \cdots \quad w^{\top} x_{N}-y_{N} \right) \left(\begin{array}{c}
{w^{\top} x_{1}-y_{1}} \\
{w^{\top} x_{2}-y_{2}} \\
{\vdots} \\
{w^{\top} x_{N}-y_{N}}
\end{array}\right) \\
&= \left(w^{\top} X^{\top} - Y^{\top} \right) \left(Xw - Y \right) \\
&= w^{T} {X}^{\top} X w-2 w^{\top} X^{\top} Y+Y^{\top} Y
\end{aligned}
$$
对上式求导，并令其为$0$：
$$\hat{w}=\arg \min L(w) $$ $$\frac{\partial L(w)}{\partial w}=2 X^{\top} X w-2 X^{\top} Y = 0$$ $$ \hat{w} = \left( X^{\top} X \right)^{-1} X^{\top} Y $$
