The policy gradient methods that we have studied so far, including REINFORCE, QAC, and , are all on-policy. The reason for this can be seen from the expression of the true gradient:
To use samples to approximate this true gradient, we must generate the action samples by following . Hence, is the behavior policy. Since is also the target policy that we aim to improve, the policy gradient methods are on-policy.
In the case that we already have some samples generated by a given behavior policy, the policy gradient methods can still be applied to utilize these samples. To do that, we can employ a technique called importance sampling. It is a general technique for estimating expected values defined over one probability distribution using some samples drawn from another distribution.
We next introduce the importance sampling technique. Consider a random variable . Suppose that is a probability distribution. Our goal is to estimate . Suppose that we have some i.i.d. samples .
Suppose the samples are not generated by . Instead, they are generated by another distribution . Can we still use these samples to approximate ? The answer is yes. can be approximated based on the importance sampling technique.
In particular, satisfies
Thus, estimating becomes the problem of estimating . Let
Since can effectively approximate , we obtain
Here, is called the importance weight. When , the importance weight is 1 and becomes . When can be sampled more frequently by but less frequently by . In this case, the importance weight, which is greater than one, emphasizes the importance of this sample.
You may ask that while is required in why do we not directly calculate using its definition ?
The answer is as follows. To use the definition, we need to know either the analytical expression of or the value of for every . However, it is difficult to obtain the analytical expression of when the distribution is represented by, for example, a neural network. It is also difficult to obtain the value of for every when is large. By contrast, merely requires the values of for some samples and is much easier to implement in practice.
The off-policy policy gradient theorem
Algorithm 10.3
Like the previous on-policy case, we need to derive the policy gradient in the off-policy case.
Suppose is the behavior policy that generates experience samples.
Our aim is to use these samples to update a target policy that can minimize the metric where is the stationary distribution under policy and is the state value under policy .
The gradient of this metric is given in the following theorem.
Theorem 10.1 (Off-policy policy gradient theorem). In the discounted case where , the gradient of is where the state distribution is where is the discounted total probability of transitioning from to under policy .
The gradient is similar to that in the on-policy case in Theorem 9.1, but there are two differences. The first difference is the importance weight. The second difference is that instead of . Therefore, we can use the action samples generated by following to approximate the true gradient. The proof of the theorem is given in the appendix.
The off-policy policy gradient is also invariant to a baseline . In particular, we have
because
To reduce the estimation variance, we select the baseline as as in A2C and obtain
The corresponding stochastic gradient-ascent algorithm is
Similar to the on-policy case,
Then, the algorithm becomes and hence
Appendix
Proof of Theorem 10.1
Since is independent of , the gradient of satisfies
According to Lemma 9.2, the expression of is where Substituting into yields
where the 3rd equation is because
By using the importance sampling technique, the above equation can be further rewritten as $$
$$
The proof is complete. The above proof is similar to that of Theorem 9.1.