（2020李宏毅）机器学习- Gradient Descent

文章目录

Review: Gradient Descent

---

$\theta^{*}=\arg \min _{\theta} L(\theta) \quad L: loss\quad function \quad \theta : parameters\\ Suppose \quad that \quad \theta\quad has\quad two\quad variables \left\{\theta_{1}, \theta_{2}\right\}$

Randomly start at $\theta^{\circ}=\left[\begin{array}{l} \theta_{1}^{0} \\ \theta_{2}^{0} \end{array}\right]$

$\begin{array}{l} \left[\begin{array}{l} \theta_{1}^{1} \\ \theta_{2}^{1} \end{array}\right]=\left[\begin{array}{c} \theta_{1}^{0} \\ \theta_{2}^{0} \end{array}\right]-\eta\left[\begin{array}{l} \partial L\left(\theta_{1}^{0}\right) / \partial \theta_{1} \\ \partial L\left(\theta_{2}^{0}\right) / \partial \theta_{2} \end{array}\right] \longrightarrow \quad \theta^{1}=\theta^{0}-\eta \nabla L\left(\theta^{0}\right)\\ \left[\begin{array}{l} \theta_{1}^{2} \\ \theta_{2}^{2} \end{array}\right]=\left[\begin{array}{l} \theta_{1}^{1} \\ \theta_{2}^{1} \end{array}\right]-\eta\left[\begin{array}{l} \partial L\left(\theta_{1}^{1}\right) / \partial \theta_{1} \\ \partial L\left(\theta_{2}^{1}\right) / \partial \theta_{2} \end{array}\right] \longrightarrow \quad \theta^{2}=\theta^{1}-\eta \nabla L\left(\theta^{1}\right) \end{array}$

---

Tuning your learning rates

---

Adagrad

---

Adagrad
$\eta^{t}=\frac{\eta}{\sqrt{t+1}} \quad g^{t}=\frac{\partial L\left(\theta^{t}\right)}{\partial w}$
$w^{t+1} \leftarrow w^{t}-\frac{\eta^{t}}{\sigma^{t}} g^{t}$

推导过程：$\begin{array}{ll} w^{1} \leftarrow w^{0}-\frac{\eta^{0}}{\sigma^{0}} g^{0} & \sigma^{0}=\sqrt{\left(g^{0}\right)^{2}} \\ w^{2} \leftarrow w^{1}-\frac{\eta^{1}}{\sigma^{1}} g^{1} & \sigma^{1}=\sqrt{\frac{1}{2}\left[\left(g^{0}\right)^{2}+\left(g^{1}\right)^{2}\right]} \\ w^{3} \leftarrow w^{2}-\frac{\eta^{2}}{\sigma^{2}} g^{2} & \sigma^{2}=\sqrt{\frac{1}{3}\left[\left(g^{0}\right)^{2}+\left(g^{1}\right)^{2}+\left(g^{2}\right)^{2}\right]} \\ \vdots \\ w^{t+1} \leftarrow w^{t}-\frac{\eta^{t}}{\sigma^{t}} g^{t} & \sigma^{t}=\sqrt{\frac{1}{t+1} \sum_{i=0}^{t}\left(g^{i}\right)^{2}} \end{array}$

Adagrad解释：

---

Stochastic Gradient Descent（随机梯度下降）

---

$\text{Pick an example }x^{n}$

$L^{n}=\left(\hat{y}^{n}-\left(b+\sum w_{i} x_{i}^{n}\right)\right)^{2} \quad \theta^{i}=\theta^{i-1}-\eta \nabla L^{n}\left(\theta^{i-1}\right)$

Feature Scaling（特征缩放）

---

特征缩放作用：

面对特征数量较多的时候，保证这些特征具有相近的尺度（无量纲化），可以使梯度下降法更快的收敛。这两张图代表数据是否均一化的最优解寻解过程（左边是未归一化的），

从这两张图可以看出，数据归一化后，最优解的寻优过程明显会变得平缓，更容易正确的收敛到最优解

怎样进行特征缩放

对红色框里面的$x_i^r$进行特征缩放，就要先求出绿框里面元素的平均值$m_i$,再求出绿框里面元素的标准差$\sigma_i$,最后代入$\frac{x_i^r-m_i}{\sigma_i}$

---

Gradient Descent Theory

---

求解$\theta^{*}=\arg \min _{\theta} L(\theta)$
梯度下降的理论推导过程：

数学基础：
Taylor Series

• Taylor series: Let$h(x)$be any function infinitely differentiable around$x=x_{0}$
  $\begin{aligned} \mathrm{h}(\mathrm{x}) &=\sum_{k=0}^{\infty} \frac{\mathrm{h}^{(k)}\left(x_{0}\right)}{k !}\left(x-x_{0}\right)^{k} \\ &=h\left(x_{0}\right)+h^{\prime}\left(x_{0}\right)\left(x-x_{0}\right)+\frac{h^{\prime \prime}\left(x_{0}\right)}{2 !}\left(x-x_{0}\right)^{2}+\ldots \end{aligned}$
  When$x$is close to$x_{0} \Rightarrow h(x) \approx h\left(x_{0}\right)+h^{\prime}\left(x_{0}\right)\left(x-x_{0}\right)$
  Multivariable Taylor Series
  $\begin{aligned} h(x, y) &=h\left(x_{0}, y_{0}\right)+\frac{\partial h\left(x_{0}, y_{0}\right)}{\partial x}\left(x-x_{0}\right)+\frac{\partial h\left(x_{0}, y_{0}\right)}{\partial y}\left(y-y_{0}\right) \\ &+\text { something related to }\left(x-x_{0}\right)^{2} \text { and }\left(y-y_{0}\right)^{2}+\ldots \ldots \end{aligned}$
  When$x$and$y$is close to$x_{0}$and$y_{0}$
  $h(x, y) \approx h\left(x_{0}, y_{0}\right)+\frac{\partial h\left(x_{0}, y_{0}\right)}{\partial x}\left(x-x_{0}\right)+\frac{\partial h\left(x_{0}, y_{0}\right)}{\partial y}\left(y-y_{0}\right)$

推导过程： Based on Taylor Series: If the red circle is small enough, in the red circle$\mathrm{L}(\theta) \approx \mathrm{L}(a, b)+\frac{\partial \mathrm{L}(a, b)}{\partial \theta_{1}}\left(\theta_{1}-a\right)+\frac{\partial \mathrm{L}(a, b)}{\partial \theta_{2}}\left(\theta_{2}-b\right)$ constant$\begin{array}{l} s=\mathrm{L}(a, b) \\ u=\frac{\partial \mathrm{L}(a, b)}{\partial \theta_{1}}, v=\frac{\partial \mathrm{L}(a, b)}{\partial \theta_{2}} \end{array}$$\mathrm{L}(\theta) \approx s+u\left(\theta_{1}-a\right)+v\left(\theta_{2}-b\right)$ Find $\theta_{1}$ and $\theta_{2}$ in the red circle $s=\mathrm{L}(a, b)$ Find $\theta_{1}$ and $\theta_{2}$ in th minimizing $L(\theta)$ minimizing $L(\theta)$ $\left(\theta_{1}-a\right)^{2}+\left(\theta_{2}-b\right)^{2} \leq d^{2}$

$\left[\begin{array}{c} \theta_{1} \\ \theta_{2} \end{array}\right]=\left[\begin{array}{l} a \\ b \end{array}\right]-\eta\left[\begin{array}{l} u \\ v \end{array}\right]=\left[\begin{array}{l} a \\ b \end{array}\right]-\eta\left[\begin{array}{c} \frac{\partial \mathrm{L}(a, b)}{\partial \theta_{1}} \\ \frac{\partial \mathrm{L}(a, b)}{\partial \theta_{2}} \end{array}\right]$