强化学习(动态规划DP)之策略迭代和价值迭代

前言

首先可看马尔科夫决策过程介绍（MDP）

策略迭代和价值迭代

策略迭代分两步

1. 策略评估：对任意策略$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$能估计出该策略带来的期望累积奖赏 (价值函数)。即给定policy为$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$得到${V_\pi }(s)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$和$q_{\pi}(s, a)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，其中 $\forall s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$。
2. 策略改进：依据策略评估得到的结果，构造出新的$\pi$好于$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$。

策略迭代

1. 策略评估

1.1解析解

已知MDP的动态特性$P\left(s^{\prime}, r \mid s, a\right) \triangleq P_{r}\left\{s_{t+1}=s$ 给定$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$得到${V_\pi }?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$,其中$\forall s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，记为${V_\pi } = {\left( \begin{array}{l} {V_\pi }({s_1})\\ {V_\pi }({s_2})\\ {V_\pi }({s_3})\\ \cdots \\ {V_\pi }({s_n}) \end{array} \right)_{|s| \times 1}}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$
由贝尔曼状态期望方程$\begin{array}{l} {V_\pi }(s) = {E_\pi }[{G_t}|{S_t} = s]\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\rm{ = }}{E_\pi }[R_{{\rm{t + 1}}}^{}{\rm{ + }}\gamma {V_\pi }{S_{{\rm{t + 1}}}}{\rm{|}}{S_t} = s]\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\rm{ = }}\sum\limits_{a \in A(s)} {\pi (a|s)} \sum\limits_{r,s$

---

我们将上面的连加式分开逐个分析,第一部分令式子为$Z?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$$Z\triangleq\sum\limits_{a \in A(s)} {\pi (a|s)} \sum\limits_{r,s$我们$r(s,a)\triangleq\sum\limits_r {r \bullet p(r|s,a)} = {E_\pi }[{R_{t + 1}}|{S_t} = s,{A_t} = a]?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$所以$Z=\sum\limits_{a \in A(s)}{\pi (a|s)} \sum\limits_{r,s$因为$Z=\sum\limits_{a \in A(s)} {\pi (a|s)} r(s,a)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$这个式子与$a?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$无关，所以我们可以记为$Z=\sum\limits_{a \in A(s)} {\pi (a|s)} r(s,a) \triangleq r_\pi (s){\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$
其中$\begin{array}{c} r_{\pi}(s)=\left(\begin{array}{c} r_{\pi}\left(s_{1}\right) \\ r_{\pi}\left(s_{2}\right) \\ r_{\pi}\left(s_{3}\right) \\ \cdots \\ r_{\pi}\left(s_{|s|}\right) \end{array}\right)_{|s| \times 1} \end{array}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$

---

第二部分令式子为$T?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$$\begin{aligned} T\triangleq\sum_{a \in A(s)} \pi(a \mid s) \sum_{r, s^{\prime}} p\left(s^{\prime}, r \mid s, a\right) \gamma V_{\pi}\left(s^{\prime}\right) &=\sum_{a \in A(s)} \pi(a \mid s) \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right) \gamma V_{\pi}\left(s^{\prime}\right) \\ &=\gamma \sum_{s^{\prime}} \sum_{a \in A(s)} \pi(a \mid s) p\left(s^{\prime} \mid s, a\right) V_{\pi}\left(s^{\prime}\right) \end{aligned}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$因为$\sum\limits_{a \in A(s)} {\pi (a|s)} p(s$与$a?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$无关，所以我们可以定义${p_\pi }(s,s$所以有$T= \gamma \sum_{s^{\prime}}{p_\pi }(s,s$
我们定义一个矩阵${P_\pi } = {[{p_\pi }(s,s$

---

结合两部分$V_\pi(s)=Z+T=r_{\pi}(s)+\gamma\sum_{s^{\prime}} {p_\pi }(s,s$我们令$s_i\triangleq s?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$,$s_j\triangleq s$。所以有$V_\pi(s_i)=Z+T=r_{\pi}(s_i)+\gamma\sum\limits_{j = 1}^{{\rm{|s|}}} {p_\pi }(s_i,s_j){V_\pi }(s_j)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$所以矩阵形式为$\begin{array}{l} {V_\pi } = {r_\pi } + \gamma {P_\pi }{V_\pi } \end{array}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$求解过程${V_\pi } - \gamma {P_\pi }{V_\pi } = {r_\pi }\\ (I - \gamma {P_\pi }){V_\pi } = {r_\pi }\\ {V_\pi } = {(I - \gamma {P_\pi })^{ - 1}}{r_\pi }?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$这是矩阵形式的解析解${V_\pi } = {(I - \gamma {P_\pi })^{ - 1}}{r_\pi }?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，复杂度较高$O(|S|^3)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，很难求解

1.2 迭代策略评估（数值解）

具体的方法数$\mathop {\lim }\limits_{k \to \infty } \{ {V_k}\} = {V_\pi }?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$
由贝尔曼方程得到迭代函数（更新规则）
${V_{k + 1}}(s) = \sum\limits_{a \in A(s)} {\pi (a|s)} \sum\limits_{r,s$
我们通过这种方式，可能会思考：

• 问题1.这个$V_k?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$会不会收敛
• 问题2.这个$V_k?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$会不会收敛的$V_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$

这个收敛性的证明，比较复杂，就此打住知道结论即可。
最初阶段，我们随机初始化$V_1?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，进行迭代更新，最终得到收敛于$V_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$的数值解。
$V_{1}=\left(\begin{array}{l} V_{1}\left(s_{1}\right) \\ V_{1}\left(s_{2}\right) \\ V_{1}\left(s_{3}\right) \\ \vdots \\ V_{1}\left(s_{|s|}\right) \end{array}\right)_{|s| \times 1}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$

2 策略改进

对某个策略的累积奖赏进行评估后，若发现它并非最优策略，则当然希望对其进行改进理想的策略应能最大化累积奖赏。

2.1 策略改进定理

思想是给定一个$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$找到$\pi$，判断$V_{\pi$和$V_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$大小。但是求出$V_{\pi$和$V_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$非时费力，所以提出了利用策略改进定理，该方法不需要求出$V_{\pi$，就可以估计他们的大小。
策略改进定理：给定的$\pi$和$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，如果$s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，${q_\pi }(s,\pi$，那么则有$s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，$V_{\pi$。
注：${q_\pi }(s,a)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$即对于一个$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$来说，我们求出$q_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，则对应的$V_\pi(s)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$就也求出来了，因为$V_\pi(s)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$和${q_\pi }(s,a)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$本身就具有对应关系，考虑到$a?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$是自由变量，$a?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$换成$\pi$，比较${q_\pi }(s,\pi$ 和${V_\pi }(s)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$大小就好。

2.2 策略改进–贪心策略

对于$\forall s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，有${V_\pi }(s) \le \mathop {\max }\limits_a {q_\pi }(s,a)={q_\pi }(s,\pi$，由策略改进定理可知$\forall s \in S?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$，$V_{\pi$。直到$\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$与$\pi$一致、不再发生变化，此时就满足了最优Bellman等式，即找到了最优策略。

价值迭代

回顾：
策略迭代分为两步

• 策略评估                $\pi→V_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$
• 策略改进                贪心策略最大化累积奖赏。

问题：因为策略迭代时迭代里面套迭代的方法，即策略评估和策略改进两个步骤均需要迭代计算，由此可知其计算还是比较慢的。
思考：在求解过程中，我们需要将$V_\pi?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$的精确解给求出吗?
$\pi→V_{\pi1}→\pi_2→V_{\pi2}→...→V_{\pi}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$其实我们并不关心这些步骤，重要的是我们要求出最优价值$V_*?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$和对应的最优策略$\pi_*?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$。
解决方案：截断策略评估（截断策略评估的迭代步骤），比如说:假设策略评估需要迭代100，我们只迭代90次。

极端情况下截断策略迭代

• 策略评估只进行一次迭代
• 策略改进

其迭代公式为：
${V_{k + 1}}(s) = \mathop {\max }\limits_a \sum\limits_{r,s$

---

我们来对上式子做一个分析：
状态-动作价值函数的迭代式为：${q_{k + 1}}(s,a) = \sum\limits_{r,s$
我们找到最优的actioon：$a_{new}=\mathop {\arg \max }\limits_a {q_{k + 1}}(s,a)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$$a_{new}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$为新的策略，我们可推：
${V_{k + 1}}(s) = \max {q_{k + 1}}(s,a)?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$
所以极端情况下的迭代公式为：
${V_{k + 1}}(s) = \mathop {\max }\limits_a \sum\limits_{r,s$

---

以上这种方法称为价值迭代
$\pi→V_{\pi1}→\pi_2→V_{\pi2}→...→V_{\pi}?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_30,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=$
价值迭代是极端情况下的策略迭代。