(可选) Policy Gradient Theorem

在这个可选章节中，我们将研究如何区分目标函数，我们将使用该目标函数来近似策略梯度。

让我们首先回顾一下不同的公式

1. 目标函数

2. 轨迹的概率（假设动作来自$\pi_\theta$):

所以我们有$\nabla_\theta J(\theta) = \nabla_\theta \sum_{\tau}P(\tau;\theta)R(\tau)$

我们可以将总和的梯度重写为梯度的总和$= \sum_{\tau} \nabla_\theta (P(\tau;\theta)R(\tau)) = \sum_{\tau} \nabla_\theta P(\tau;\theta)R(\tau)$因为$R(\tau)$不依赖于$\theta$

然后我们将总和中的每一项乘以$\frac{P(\tau;\theta)}{P(\tau;\theta)}$（这是可能的，因为它 = 1）$= \sum_{\tau} \frac{P(\tau;\theta)}{P(\tau;\theta)}\nabla_\theta P(\tau;\theta)R(\tau)$

我们可以进一步简化，因为$\frac{P(\tau;\theta)}{P(\tau;\theta)}\nabla_\theta P(\tau;\theta) = P(\tau;\theta)\frac{\nabla_\theta P(\tau;\theta)}{P(\tau;\theta)}$.

因此，我们可以将总和重写为$P(\tau;\theta)\frac{\nabla_\theta P(\tau;\theta)}{P(\tau;\theta)}= \sum_{\tau} P(\tau;\theta) \frac{\nabla_\theta P(\tau;\theta)}{P(\tau;\theta)}R(\tau)$

然后我们可以使用导数对数技巧（也称为 似然比技巧 或 REINFORCE 技巧），这是微积分中的一个简单规则，意味着$\nabla_x log f(x) = \frac{\nabla_x f(x)}{f(x)}$

所以给定我们有$\frac{\nabla_\theta P(\tau;\theta)}{P(\tau;\theta)}$we transform it as$\nabla_\theta log P(\tau|\theta)$

So this is our likelihood policy gradient$\nabla_\theta J(\theta) = \sum_{\tau} P(\tau;\theta) \nabla_\theta log P(\tau;\theta) R(\tau)$

Thanks for this new formula, we can estimate the gradient using trajectory samples (we can approximate the likelihood ratio policy gradient with sample-based estimate if you prefer).$\nabla_\theta J(\theta) = \frac{1}{m} \sum^{m}_{i=1} \nabla_\theta log P(\tau^{(i)};\theta)R(\tau^{(i)})$where each$\tau^{(i)}$is a sampled trajectory.

But we still have some mathematics work to do there: we need to simplify$\nabla_\theta log P(\tau|\theta)$

We know that$\mu(s_0)$