【李航】统计学习方法--4. 朴素贝叶斯法（详细推导）

文章目录

4.1朴素贝叶斯法的学习与分类

---

贝叶斯定理

---

• 贝叶斯思维



• 条件概率
  ​$P(X=x \mid Y=y)=\frac{P(X=x, Y=y)}{P(Y=y)}$
• 贝叶斯定理
  已知：
  存在 $K$ 类 $c_{1}, c_{2}, \cdots, c_{K}$ , 给定一个新的实例 $x=\left(x^{(1)}, x^{(2)}, \cdots, x^{(n)}\right)$
  问：该实例归属第 $c_{i}$ 类的可能性有多大?
  ​$P\left(Y=c_{i} \mid X=x\right)=\frac{P\left(X=x \mid Y=c_{i}\right) \cdot P\left(Y=c_{i}\right)}{P(X=x)}$
  即，
  ​$P\left(Y=c_{i} \mid X=x\right)=\frac{P\left(X=x \mid Y=c_{i}\right) \cdot P\left(Y=c_{i}\right)}{\sum_{i=1}^{K} P\left(X=x \mid Y=c_{i}\right) \cdot P\left(Y=c_{i}\right)}$
  ​$\arg \max _{c_{i}} P\left(X=x \mid Y=c_{i}\right) \cdot P\left(Y=c_{i}\right)$
• 朴素贝叶斯
  假设：实例特征之间相互独立
  $P\left(X=x \mid Y=c_{i}\right)=\prod_{j=1}^{n} P\left(X^{(j)}=x^{(j)} \mid Y=c_{i}\right) \\\Longrightarrow P(X=x)=\sum_{i=1}^{K} P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X^{(j)}=x^{(j)} \mid Y=c_{i}\right) \\\Longrightarrow P\left(Y=c_{i} \mid X=x\right)=\frac{P\left(X=x \mid Y=c_{i}\right) \cdot P\left(Y=c_{i}\right)}{\sum_{i=1}^{K}P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X(j)=x^{(j)} \mid Y=c_{i}\right)} \\\Longrightarrow P\left(Y=c_{i} \mid X=x\right)=\frac{P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X^{(j)}=x^{(j)} \mid Y=c_{i}\right)}{\sum_{i=1}^{K} P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X(j)=x^{(j)} \mid Y=c_{i}\right)}$
  $\underset{c_{i}}{\arg \max } P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X^{(j)}=x^{(j)} \mid Y=c_{i}\right)$

4.1.1 基本方法

---

• 训练数据集:
  $T=\left\{\left(x_{1}, y_{1}\right),\left(x_{2}, y_{2}\right) \cdots,\left(x_{N}, y_{N}\right)\right\}$

• 输入:$\mathcal{X} \subseteq \mathbf{R}^{n}, x \in \mathcal{X}$
• 输出:$\mathcal{Y}=\left\{c_{1}, c_{2}, \cdots, c_{K}\right\}, y \in \mathcal{Y}$
  生成方法：学习联合概率分布$P(X, Y)$

• 生成方法：学习联合概率分布$P(X, Y)$

• 先验概率分布：

$P\left(Y=c_{i}\right), \quad i=1,2, \cdots, K$

• 条件概率分布:

$P\left(X=x \mid Y=c_{i}\right)=P\left(X^{(1)}=x^{(1)}, \cdots, X^{(n)}=x^{(n)} \mid Y=c_{i}\right)$

• 联合概率分布：

$P(X, Y)=P\left(X=x \mid Y=c_{i}\right) P\left(Y=c_{i}\right), i=1,2, \cdots, K$
假设是独立的是为了能够计算出来，使其具有可行性

4.1.2 后验概率最大化的含义

---

• 后验概率
  ​$P\left(Y=c_{i} \mid X=x\right)=\frac{P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X^{(j)}=x^{(j)} \mid Y=c_{i}\right)}{\sum_{i=1}^{K} P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X(j)=x^{(j)} \mid Y=c_{i}\right)}$
• 朴素贝叶斯法将实例分到后验概率最大的类中。这等价于期望风险最小化。假设选择$0-1$损失函数:
  $L ( Y , f ( X ) ) = \{ \begin{array} { l l } { 1 , } & { Y \neq f ( X ) } \\ { 0 , } & { Y = f ( X ) } \end{array}$
  式中$f(X)$是分类决策函数。这时,期望风险函数为
  $R _ { e x p } ( f ) = E [ L ( Y , f ( X ) ) ]$
  因为期望的定义是值出现的概率乘以具体值之和，所以上式可转换为损失函数与联合概率之积的积分：
  $R_{\exp }(f)=E[L(Y, f(X))] \\=\int_{\chi \times y} L(y, f(x)) P(x, y) d x d y \\=\int_{\chi \times y} L(y, f(x)) P(y \mid x) P(x) d x d y \\=\int_{\chi} \int_{Y} L(y, f(x)) P(y \mid x) d y P(x) d x$
  期望是对联合分布$P(X,Y)$取的。由此取条件期望
  $R _ { e x p } ( f ) = E x \sum _ { k = 1 } ^ { K } [ L ( c _ { k } , f ( X ) ) ] P ( c _ { k } | X )$
  为了使期望风险最小化,只需对$X=x$逐个极小化,由此得到:
• $\begin{aligned} f(x) &=\arg \min _{y \in \mathcal{Y}} \sum_{k=1}^{K} L\left(c_{k}, y\right) P\left(c_{k} \mid X=x\right) \\ &=\arg \min _{y \in \mathcal{Y}} \sum_{k=1}^{K} P\left(y \neq c_{k} \mid X=x\right) \\ &=\arg \min _{y \in \mathcal{Y}}\left(1-P\left(y=c_{k} \mid X=x\right)\right) \\ &=\arg \max _{y \in \mathcal{Y}} P\left(y=c_{k} \mid X=x\right) \end{aligned}$
• 这样一来，根据期望风险最小化准则就得到了后验概率最大化准则:
  ​$f ( x ) = \operatorname { a r g m a x } P ( c _ { k } | X = x )$
  即朴素贝叶斯法所采用的原理.

4.2 朴素贝叶斯法的参数估计

---

4.2.1 极大似然估计

---

• 由$\underset{c_{i}}{\arg \max } P\left(Y=c_{i}\right) \prod_{j=1}^{n} P\left(X^{(j)}=x^{(j)} \mid Y=c_{i}\right)$可知，学习意味着估计$P ( Y = c _ { k } )$和$P ( X ^ { ( j ) } = x ^ { ( j ) } | Y = c _ { k } )$
• 极大似然估计

1. $P ( Y = c _ { k } ) = \frac { \sum _ { i = 1 } ^ { N } I ( y _ { i } = c _ { k } ) } { N } , \quad k = 1 , 2 , \cdots , K$$N$是样本，分子是点的个数
2. 设第$j$个特征$x^{(j)}$可能取值的集合为$\left\{a_{j 1}, a_{j 2}, \cdots, a_{j S_{j}}\right\}$, 条件概率$P\left(X^{(j)}=a_{j l} \mid Y=\right. \left.c_{k}\right)$的极大似然估计是
   $P\left(X^{(j)}=a_{j l} \mid Y=c_{k}\right)=\frac{\sum_{i=1}^{N} I\left(x_{i}^{(j)}=a_{j l}, y_{i}=c_{k}\right)}{\sum_{i=1}^{N} I\left(y_{i}=c_{k}\right)} \\j=1,2, \cdots, n ; \quad l=1,2, \cdots, S_{j} ; \quad k=1,2,\cdots,K$
   式中,$x_{i}^{(j)}$是第$i$个样本的第$j$个特征;$a_{j l}$是第$j$个特征可能取的第$l$个值;$I$

4.2.2 学习与分类算法

---

1. 计算先验概率及条件概率
   $P ( Y = c _ { k } ) = \frac { \sum _ { i = 1 } ^ { N } I ( y _ { i } = c _ { k } ) } { N } , \quad k = 1 , 2 , \cdots , K$
   $P\left(X^{(j)}=a_{j l} \mid Y=c_{k}\right)=\frac{\sum_{i=1}^{N} I\left(x_{i}^{(j)}=a_{j l}, y_{i}=c_{k}\right)}{\sum_{i=1}^{N} I\left(y_{i}=c_{k}\right)} \\j=1,2, \cdots, n ; \quad l=1,2, \cdots, S_{j} ; \quad k=1,2,\cdots,K$
2. 对于给定实例$x = ( x ^ { ( 1 ) } , x ^ { ( 2 ) } , \cdots , x ^ { ( n ) } ) ^ { T }$，计算
   ​$P ( Y = c _ { k } ) \prod _ { j = 1 } ^ { n } P ( X ^ { ( j ) } ) Y = x ^ { ( j ) } | Y = c _ { k } ) , \quad k = 1 , 2 , \cdots , K$
3. 确定实例的类
   $y = \operatorname { a r g m a x } P ( Y = c _ { k } ) \prod _ { j = 1 } ^ { n } P ( X ^ { ( j ) } ) | Y = c _ { i } ) | Y = c _ { k } )$

4.2.3 贝叶斯估计

---

• 先验概率的贝叶斯估计

$P_{\lambda}\left(Y=c_{k}\right)=\frac{\sum_{i=1}^{N} I\left(y_{i}=c_{k}\right)+\lambda}{N+K \lambda}$

• 条件概率的贝叶斯估计

$P_{\lambda}\left(X^{(j)}=a_{j l} \mid Y=c_{k}\right)=\frac{\sum_{i=1}^{N} I\left(x_{i}^{(j)}=a_{j l}, y_{i}=c_{k}\right)+\lambda}{\sum_{i=1}^{N} I\left(y_{i}=c_{k}\right)+S_{j} \lambda}$
注： $\lambda \geq 0 \quad \lambda=0$时为极大似然估计, $\lambda=1$ 时为拉普拉斯平滑(Laplacian Smoothing)。
为什么+$K$$\lambda$?
​$\sum _ { k = 1 }^{K} P _ { k } ( Y = C _ { k } ) = 1$
为什么+$S_{j} \lambda$?
​$\sum _ { l = 1 }^{S_{j}} P_{\lambda}\left(X^{(j)}=a_{j l} \mid Y=c_{k}\right)=1$