陆昱宽

This chapter introduces Q learning, a Temporal-Difference (TD) learning method to estimate optimal action values, hen optimal policies. Previouly we have illustrated TD-learning of state values and action values. For these methods, we need to do policy improvement to get optimal policies.

Sources:

1. Shiyu Zhao. Chapter 7: Temporal-Difference Methods. Mathematical Foundations of Reinforcement Learning.

In the previous post, we introduced TD learning algorithms. At that time, all state/action values were represented by tables. This is inefficient for handling large state or action spaces.

In this post, we will use the function approximation method for TD learning. It is also where artificial neural networks are incorporated into reinforcement learning as function approximators.

Sources:

1. Shiyu Zhao. Chapter 8: Value Function Approximation. Mathematical Foundations of Reinforcement Learning.

This post introduces TD learning--the kind of value function learning that leverages a bootstrapped target value, called TD target.

Sources:

1. Shiyu Zhao. Chapter 7: Temporal-Difference Methods. Mathematical Foundations of Reinforcement Learning.
2. Modern world model agents that adopts TD-learning:
   1. DreamerV3 paper
   2. DIAMOND paper
   3. TD-MPC paper

Sources:

1. B. P. Lathi & Roger Green. (2018). Chapter 7: Continuous-Time Signal Analysis: The Fourier Transform. Signal Processing and Linear Systems (3nd ed., pp. 701-720). Oxford University Press.

For a quick reference table, see Wikipedia page on the Fourier transform.

For more results, refer to Table of the Laplace Transform Pairs.

Sources:

1. B. P. Lathi & Roger Green. (2018). Chapter 1: Signals and Systems. Signal Processing and Linear Systems (3rd ed., pp. 97-107). Oxford University Press.

Sources:

B. P. Lathi & Roger Green. (2018). Chapter 7: Continuous-Time Signal Analysis. Signal Processing and Linear Systems (3nd ed., pp. 700-701). Oxford University Press.

Sources:

B. P. Lathi & Roger Green. (2018). Chapter 6: Continuous-Time Signal Analysis: The Fourier Series. Signal Processing and Linear Systems (3nd ed., pp. 612-620). Oxford University Press.

Sources:

B. P. Lathi & Roger Green. (2018). Chapter 7: Continuous-Time Signal Analysis. Signal Processing and Linear Systems (3nd ed., pp. 699-708). Oxford University Press.

Sources:

B. P. Lathi & Roger Green. (2018). Chapter 7: Continuous-Time Signal Analysis. Signal Processing and Linear Systems (3nd ed., pp. 704-708). Oxford University Press.