由于高维映射使得VC维增加，此时，若数据样本个数N有限时，就容易造成过拟合(Overfitting)。

1. What is Overfitting?

Overfitting 产生的原因(老师举了驾车的例子)：

2. The Role of Noise and Data Size

当数据样本有限（甚至比较少）时，简单模型比复杂模型性能更优！

3. Deterministic Noise

两类误差：1）数据样本的误差；2）模型表示能力不足带来的误差，如50维的多项式用一个20维的多项式去表示，造成表示不足的误差！

Deterministic Noise

4. Dealing with Overfitting

All very practical techniques to combat overfitting！

• Correct the label (data cleaning);

• Remove the example (data pruning);

    Possibly helps, but effect varies!

• Add virtual examples by shifting/rotating the given digits (data hinting);

   Possibly helps, but watch out —virtual example not i.i.d. $\sim$ $P(x; y)$!