---

前言

本文为7月22日深度学习笔记，分为五个章节：

• Step 1: Model；
• Step 2: Goodness of Function；
• Step 3: Find Best Function(Gradient Descent)；
• Back to Step 1: Redesign；
• Back to Step 2: Regularization 正则化。

---

一、Step 1: Model

Estimating the combat power (CP) of a pokemon after evolution

$$y=b+w\cdot x_{cp}.$$

• w: weight;
• b: bias.

---

二、Step 2: Goodness of Function

• A set of function: $f_1,f_2\dots \dots$
• function input: $x^1,x^2\dots \dots$
• function output: ${\hat{y}}_1,{\hat{y}}_2\dots \dots$

Training Data:
$$(x^1,{\hat{y}}_1),(x^2,{\hat{y}}_2),\dots ,(x^{1}0,{\hat{y}}_{1}0)$$

• Loss function $L$：
• Input: a function
• output: how bad it is
  $$L(f)=\sum _{n=1}^{10}({\hat{y}}^n-f(x_{cp}^n){)}^2$$
  Where $f(x_{cp}^n)$ is estimated $y$ based on input function.
  $L(f)$ also equals:
  $$L(w,b)=\sum _{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2$$

Each point in the figure is a function.

---

三、Step 3: Find Best Function(Gradient Descent)

How to pick the best function: gradient descent

$$f\ast =argminL(f)$$
$$w\ast ,b\ast =argminL(w,b)=argmin\sum _{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2$$

• 首先考虑1个参数 $w$ 的情况：
  1. 取初始 $w^0$；
  2. 在该点求导：$\frac{dL}{dw}{|}_{w=w^0}$
  3. 若为负，向右移动 $w$；
  4. 若为正，向左移动 $w$；
  5. $w^0-\eta \frac{dL}{dw}{|}_{w=w^0}\to w^1$, $\eta$ is “learning rate”；
  6. $w^1-\eta \frac{dL}{dw}{|}_{w=w^1}\to w^2$，以此类推；

• 当有2个参数 $w,b$ 的情况：
  1. $gradient=\frac{\partial L}{\partial w},\frac{\partial L}{\partial b}$；
  2. $w^0-\eta \frac{\partial L}{\partial w}{|}_{w=w^0}\to w^1$；
  3. $b^0-\eta \frac{\partial L}{\partial b}{|}_{b=b^0}\to b^1$

• **Results: **
  $$AverageErroronTestingData=\frac{1}{10}\sum _{n=1}^{10}{e}^n$$

• Model Selection:
  
  但是，a more complex model does not always lead to better performance, which is Overfitting

---

四、 Back to Step 1: Redesign

• Find the hidden factors.
  

$$\begin{gathered} x_s=speciesofx \\ \delta (x_s=Pidgey)\{\begin{array}{l}=1if{x}_s=Pidgey\\ =0otherwise\end{array} \end{gathered}$$

---

五、Back to Step 2: Regularization 正则化

$$\begin{gathered} y=b+\sum w_i{x}_i \\ L=\sum _n({\hat{y}}^n-y{)}^2+\lambda \sum (w_i{)}^2 \end{gathered}$$

• Smaller $w_i$ means smoother；
• But don’t be too smooth.

---

六、Error

来源：

• bias；
• variance.

Bias and Variance of Esitimator:
A variable x:

• $\mu$: the mean of x;
• ${\sigma }^2$: the variance（方差）of x;
• Estimator（期望） of $\mu$:
  $$\begin{gathered} m=\frac{1}{N}\sum _n{x}^n\ne \mu \\ E[m]=E[\frac{1}{N}\sum _n{x}^n]=\frac{1}{N}\sum _{n}E[x^n]=\mu \end{gathered}$$
• Variance depends on the number of samples:
  $$\begin{gathered} Var[m]=\frac{{\sigma }^2}{N} \\ s=\frac{1}{N}\sum _n(x^n-m{)}^2 \end{gathered}$$
• Biased estimator 偏差估计量：
  $$\begin{gathered} s=\frac{1}{N}\sum _n(x^n-m{)}^2 \\ E[s]=\frac{N-1}{N}{\sigma }^2\ne sigm{a}^2 \end{gathered}$$

• What to do with large bias:
  1. model cannot fit the training examples ⇒ Uderfitting, large bias ⇒ add more features + a more complex model
  2. model fit the training examples, but large on testing data ⇒ Overfitting, large variance ⇒ more data(not always practical) + regularization(may increase bias)