This post introduces TD learning–the kind of value function learning that leverages a bootstrapped target value, called TD target.

Sources:

1. Shiyu Zhao. Chapter 7: Temporal-Difference Methods. Mathematical Foundations of Reinforcement Learning.
2. Modern world model agents that adopts TD-learning:
   1. DreamerV3 paper
   2. DIAMOND paper
   3. TD-MPC paper

Temporal-Difference Methods

Temporal-difference (TD) methods are a central family of reinforcement learning algorithms. Like Monte Carlo (MC) learning, TD learning can be used for policy evaluation: given an environment and a policy \(\pi\), estimate the state-value function \(v_\pi\) or the action-value function \(q_\pi\) of that policy.

The goal of (state or action) value function learning is to estimate the expected return. For example, the state-value function of policy \(\pi\) is, we use state-value function case:

\[ v_\pi(s) = \mathbb{E}_\pi[G_t\mid S_t=s], \]

where

\[ G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots \]

is the discounted return.

A direct way to learn a value function is to use the full sampled return as the target. This is the Monte Carlo idea. If a trajectory visits \(s_t\), then MC learning updates the value estimate \(v_t(s_t)\) toward

\[ G_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots. \]

So the MC target (the target we want the value function to apprixomate) is

\[ \text{MC target} = G_t. \]

However, this requires waiting for a full return, which can be long, high-variance, or even unavailable in continuing tasks. A natural alternative is to use only a short prefix of the future trajectory and approximate the remaining future return by a learned estimate. This idea is called bootstrapping.

In general, bootstrapping means using an existing learned estimate inside a new training target. In RL value function learning, the learned estimate used for bootstrapping is usually the value function that we are trying to train. For example, instead of using the full return after \(s_t\), we can use

\[ r_{t+1} + \gamma v_t(s_{t+1}). \]

Here, \(r_{t+1}\) is an observed reward, while \(v_t(s_{t+1})\) is a learned estimate of the remaining future return starting from \(s_{t+1}\).

Therefore, in RL’s value function learning, bootstrapping constructs a value target of the form

\[ \text{value target} = \text{observed reward prefix} + \text{bootstrapped future value}. \]

In the context of TD learning, this bootstrapped value target is called a TD target. The simplest example is the one-step TD target:

\[ \bar v_t = r_{t+1} + \gamma v_t(s_{t+1}). \]

TD learning is the family of value-learning methods that update value estimates using such TD targets. In other words,

\[ \text{TD learning} = \text{value function learning with bootstrapped value targets}. \]

There are several ways to construct TD targets. Two common ones are:

• \(n\)-step TD target: use \(n\) observed rewards before bootstrapping;
• \(\lambda\)-return: mix different \(n\)-step TD targets with exponentially decaying weights.

Thus, \(n\)-step TD targets and \(\lambda\)-returns are not separate ideas from TD learning. They are different ways of constructing TD targets.

Finally, TD learning is often written not directly in terms of the TD target, but in terms of the difference between the TD target and the current value estimate. This difference is called the TD error:

\[ \delta_t = \bar v_t - v_t(s_t). \]

For the one-step TD target, TD error is

\[ \delta_t = r_{t+1} + \gamma v_t(s_{t+1}) - v_t(s_t). \]

which measures how much the current value estimate disagrees with the bootstrapped target.

The main conceptual chain is therefore:

\[ \text{bootstrapping} \rightarrow \text{TD target} \rightarrow \text{TD error} \rightarrow \text{TD update}. \]

Motivation: estimating returns without full rollouts

In policy evaluation, we want to estimate the value of a policy \(\pi\). The state-value function is

\[ v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t=s], \]

where the discounted return is

\[ G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots. \]

If we use MC learning, then after visiting \(s_t\), we wait until the episode terminates and compute the full sampled return

\[ G_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots. \]

Then we update \(v_t(s_t)\) toward \(G_t\).

This is conceptually simple, but it has several limitations:

• We must wait until the full return is available.
• Long-horizon returns can have high variance.
• In continuing tasks, the trajectory may not terminate naturally.
• In world-model-based RL, rolling out a learned model too far can cause compounding model error.

TD learning uses a different idea. It replaces the full future return after some finite horizon by the current estimate of a value function. The simplest case is

\[ G_t \approx r_{t+1} + \gamma v_t(s_{t+1}). \]

This approximation is called bootstrapping: the target for \(v_t(s_t)\) depends on another value estimate \(v_t(s_{t+1})\).

Thus, the TD idea can be summarized as:

\[ \text{estimate long-term return} \approx \text{finite observed rewards} + \text{estimated remaining return}. \]

This idea is the basis of one-step TD, \(n\)-step TD, \(\lambda\)-returns, Sarsa, Q-learning, actor-critic methods, and many modern world-model RL algorithms.

Mean estimation and the Robbins-Monro view

We first recall some stochastic approximation examples and show how TD learning can be viewed as a Robbins-Monro-like update.

Estimating a mean

Consider the problem of estimating

\[ w = \mathbb{E}[X] \]

from iid samples \(\{x_k\}\) of \(X\).

Define

\[ g(w) = w-\mathbb{E}[X]. \]

Solving \(g(w)=0\) gives the desired mean. Since we only have samples, we use the noisy observation

\[ \tilde g(w,\eta) = w-x = (w-\mathbb{E}[X]) + (\mathbb{E}[X]-x) \doteq g(w)+\eta. \]

The Robbins-Monro update is

\[ w_{k+1} = w_k - \alpha_k \tilde g(w_k,\eta_k) = w_k - \alpha_k(w_k-x_k). \]

Equivalently,

\[ w_{k+1} = w_k + \alpha_k(x_k-w_k). \]

So the estimate is moved toward the new sample \(x_k\).

Estimating the mean of a function

Now suppose we want to estimate

\[ w = \mathbb{E}[v(X)] \]

from samples \(\{x_k\}\) of \(X\).

Define

\[ g(w) = w-\mathbb{E}[v(X)]. \]

A noisy observation is

\[ \tilde g(w,\eta) = w-v(x) = (w-\mathbb{E}[v(X)]) + (\mathbb{E}[v(X)]-v(x)) \doteq g(w)+\eta. \]

The corresponding update is

\[ w_{k+1} = w_k - \alpha_k[w_k-v(x_k)]. \]

Equivalently,

\[ w_{k+1} = w_k + \alpha_k[v(x_k)-w_k]. \]

Estimating a bootstrapped target

Now consider

\[ w = \mathbb{E}[R+\gamma v(X)], \]

where \(R\) and \(X\) are random variables, \(\gamma\) is a constant, and \(v(\cdot)\) is a function.

Given samples \(r_k\) and \(x_k\), define

\[ g(w) = w-\mathbb{E}[R+\gamma v(X)]. \]

A noisy observation is

\[ \tilde g(w,\eta) = w-[r+\gamma v(x)]. \]

Then the Robbins-Monro-style update is

\[ w_{k+1} = w_k - \alpha_k \left[ w_k - (r_k+\gamma v(x_k)) \right]. \]

Equivalently,

\[ w_{k+1} = w_k + \alpha_k \left[ r_k+\gamma v(x_k)-w_k \right]. \]

This already has the form of TD learning. The target is not a full return. It is a one-step reward plus a bootstrapped value estimate.

Bellman expectation equation

Recall that the state-value function of policy \(\pi\) is

\[ v_\pi(s) = \mathbb{E}_\pi[G_t \mid S_t=s], \quad s\in\mathcal{S}. \]

Since

\[ G_t = R_{t+1} + \gamma G_{t+1}, \]

we have

\[ v_\pi(s) = \mathbb{E}_\pi [ R_{t+1} + \gamma G_{t+1} \mid S_t=s ]. \]

By the definition of \(v_\pi\), the expected future return from the next state is

\[ \mathbb{E}_\pi[G_{t+1}\mid S_{t+1}] = v_\pi(S_{t+1}). \]

Therefore,

\[ v_\pi(s) = \mathbb{E}_\pi [ R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s ]. \]

Equivalently, in transition-probability notation,

\[ v_\pi(s) = \sum_a \pi(a\mid s) \sum_{s',r} p(s',r\mid s,a) [ r+\gamma v_\pi(s') ]. \]

This is the Bellman expectation equation for state values.

Similarly, for action values,

\[ q_\pi(s,a) = \mathbb{E}_\pi [ R_{t+1} + \gamma q_\pi(S_{t+1},A_{t+1}) \mid S_t=s,A_t=a ]. \]

Equivalently,

\[ q_\pi(s,a) = \sum_{s',r} p(s',r\mid s,a) \left[ r+ \gamma \sum_{a'} \pi(a'\mid s') q_\pi(s',a') \right]. \]

These Bellman equations are the theoretical foundation of TD methods. They show that the value of the current state can be expressed in terms of the reward and the value of the next state.

From Bellman equation to TD learning

The Bellman expectation equation says

\[ v_\pi(s_t) = \mathbb{E}_\pi [ R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s_t ]. \]

A natural sample-based update is to move \(v_t(s_t)\) toward the sample target

\[ r_{t+1} + \gamma v_\pi(s_{t+1}). \]

This would give

\[ v_{t+1}(s_t) = v_t(s_t) - \alpha_t(s_t) \left[ v_t(s_t) - (r_{t+1}+\gamma v_\pi(s_{t+1})) \right]. \]

However, \(v_\pi(s_{t+1})\) is unknown. Therefore, TD learning replaces the true value \(v_\pi(s_{t+1})\) by the current estimate \(v_t(s_{t+1})\):

\[ v_{t+1}(s_t) = v_t(s_t) - \alpha_t(s_t) \left[ v_t(s_t) - (r_{t+1}+\gamma v_t(s_{t+1})) \right]. \]

Equivalently,

\[ v_{t+1}(s_t) = v_t(s_t) + \alpha_t(s_t) \left[ r_{t+1} + \gamma v_t(s_{t+1}) - v_t(s_t) \right]. \]

This is the TD(0) update.

The phrase temporal difference comes from the difference between the current value estimate and a target built from the next time step:

\[ r_{t+1} + \gamma v_t(s_{t+1}) - v_t(s_t). \]

This is the temporal-difference error.

TD learning of state values

Given data generated by a policy \(\pi\),

\[ (s_0,r_1,s_1,\ldots,s_t,r_{t+1},s_{t+1},\ldots), \]

or equivalently the transition samples

\[ \{(s_t,r_{t+1},s_{t+1})\}_t, \]

the TD(0) algorithm for estimating \(v_\pi\) is

\[ v_{t+1}(s_t) = v_t(s_t) + \alpha_t(s_t) \delta_t, \]

where

\[ \delta_t \doteq r_{t+1} + \gamma v_t(s_{t+1}) - v_t(s_t) \]

is the TD error.

Equivalently,

\[ v_{t+1}(s_t) = v_t(s_t) - \alpha_t(s_t) \left[ v_t(s_t) - \left( r_{t+1} + \gamma v_t(s_{t+1}) \right) \right]. \]

For all unvisited states,

\[ v_{t+1}(s) = v_t(s), \quad \forall s\neq s_t. \]

The quantity

\[ \bar v_t \doteq r_{t+1} + \gamma v_t(s_{t+1}) \]

is called the TD target.

Thus,

\[ \delta_t = \bar v_t - v_t(s_t). \]

At each time step, TD learning updates only the value of the visited state \(s_t\). The value estimates of other states remain unchanged.

One-step TD target and TD error

Take one-step TD learning as an example.

The one-step TD target uses one observed reward before bootstrapping:

\[ \bar v_t^{(1)} = r_{t+1} + \gamma v_t(s_{t+1}). \]

The TD error is defined as the TD target minus the current value estimate:

\[ \delta_t = \bar v_t^{(1)} - v_t(s_t). \]

Substituting the one-step TD target gives

\[ \delta_t = r_{t+1} + \gamma v_t(s_{t+1}) - v_t(s_t). \]

The TD update can be written using the TD target:

\[ v_{t+1}(s_t) = v_t(s_t) + \alpha_t(s_t) \left[ \bar v_t^{(1)} - v_t(s_t) \right]. \]

Equivalently, it can be written using the TD error:

\[ v_{t+1}(s_t) = v_t(s_t) + \alpha_t(s_t)\delta_t. \]

So, in practice, TD learning can be described in two equivalent ways:

\[ \text{update toward the TD target} \]

or

\[ \text{update by the TD error}. \]

The TD target gives the desired direction of learning, and the TD error tells us how far the current value estimate is from that target.

\(n\)-step TD target

The one-step TD target bootstraps after one observed reward. More generally, an \(n\)-step TD target uses \(n\) observed rewards before bootstrapping:

\[ \bar v_t^{(n)} = r_{t+1} + \gamma r_{t+2} + \cdots + \gamma^{n-1}r_{t+n} + \gamma^n v_t(s_{t+n}). \]

Equivalently,

\[ \bar v_t^{(n)} = \sum_{k=0}^{n-1} \gamma^k r_{t+k+1} + \gamma^n v_t(s_{t+n}). \]

The structure is:

\[ \text{first } n \text{ observed rewards} + \text{bootstrapped value at } s_{t+n}. \]

When \(n=1\), this reduces to the one-step TD target:

\[ \bar v_t^{(1)} = r_{t+1} + \gamma v_t(s_{t+1}). \]

When \(n\) is large enough to reach the end of the episode, the target becomes close to the Monte Carlo target.

Thus, \(n\) controls the bias-variance tradeoff:

• smaller \(n\): more bootstrapping, usually lower variance but higher bias;
• larger \(n\): less bootstrapping, usually lower bias but higher variance.

\(\lambda\)-return

Instead of choosing a single prefix length \(n\), we can mix different \(n\)-step TD targets. This gives anthoer contruction of TD target and we call it the \(\lambda\)-return.

The \(\lambda\)-return is a weighted mixture of different \(n\)-step TD targets. \[ G_t^\lambda = (1-\lambda) \sum_{n=1}^{\infty} \lambda^{n-1} \bar v_t^{(n)}. \]

Here, each \(\bar v_t^{(n)}\) is an \(n\)-step TD target:

\[ \bar v_t^{(n)} = \sum_{k=0}^{n-1} \gamma^k r_{t+k+1} + \gamma^n v_t(s_{t+n}). \]

The parameter \(\lambda\in[0,1]\) controls how much weight is put on longer returns:

• \(\lambda=0\): only the one-step TD target is used.
• \(\lambda\) close to \(1\): longer TD targets receive more weight.
• \(\lambda=1\): in episodic tasks, the target approaches the Monte Carlo return.

\(\lambda\)-return in finite-horizon case

In finite-horizon settings, \(\lambda\)-returns are often written recursively. Suppose we have a trajectory segment of length \(H\). A common recursive form is \[ \Lambda_t = r_{t+1} + \gamma(1-d_{t+1}) \left[ (1-\lambda)v_t(s_{t+1}) + \lambda \Lambda_{t+1} \right], \]

with terminal condition

\[ \Lambda_H = v_t(s_H). \]

Here, \(d_{t+1}\) is a termination indicator. If the episode terminates, the future value should not be bootstrapped.

This recursive form says that the future part of the target is a mixture of:

\[ v_t(s_{t+1}), \]

which is an immediate bootstrap, and

\[ \Lambda_{t+1}, \]

which continues to use a longer bootstrapped return.

TD learning of action values

In previous sections, we considered bootstrapping based on a learned state-value function. The same idea naturally extends to a learned action-value function, also called a Q-function. It is defined as

\[ q_\pi(s,a) = \mathbb{E}_\pi[G_t \mid S_t=s,A_t=a]. \]

It estimates the expected return after first taking action \(a\) in state \(s\) and then following policy \(\pi\).

For state-value TD learning, the one-step TD target is

\[ \bar v_t = r_{t+1} + \gamma v_t(s_{t+1}). \]

For action-value TD learning, the corresponding one-step target has the general form

\[ \bar q_t = r_{t+1} + \gamma q_t(s_{t+1},a_{\text{target}}). \]

The key new issue is that \(q_t\) depends on both a state and an action. After reaching the next state \(s_{t+1}\), we must decide which next action should be used in the bootstrap term. That action is denoted by

\[ a_{\text{target}}. \]

Different choices of \(a_{\text{target}}\) lead to different action-value TD algorithms.

Behavior action and target action

We distinguish two kinds of actions.

The behavior action is the action actually selected by the behavior policy and applied to the environment. It is the action that generates the trajectory.

The target action is the action used inside the TD target to compute the bootstrapped value.

They can be the same, but they do not have to be the same. Different choices give different action-value TD methods.

Sarsa: target action equals behavior action

The most direct choice is to use the action that is actually selected at the next state.

Suppose the agent follows policy \(\pi\). At time \(t\), it is in state \(s_t\), takes action \(a_t\), receives reward \(r_{t+1}\), reaches the next state \(s_{t+1}\), and then selects the next action \(a_{t+1}\) according to the same policy:

\[ a_{t+1}\sim \pi(\cdot\mid s_{t+1}). \]

If we set

\[ a_{\text{target}} = a_{t+1}, \]

then the TD target becomes

\[ \bar q_t^{\text{Sarsa}} = r_{t+1} + \gamma q_t(s_{t+1},a_{t+1}). \]

This gives the algorithm called Sarsa. The name comes from the tuple used in each update:

\[ (S_t,A_t,R_{t+1},S_{t+1},A_{t+1}). \]

The TD error is

\[ \delta_t = \bar q_t^{\text{Sarsa}} - q_t(s_t,a_t). \]

Substituting the target gives

\[ \delta_t = r_{t+1} + \gamma q_t(s_{t+1},a_{t+1}) - q_t(s_t,a_t). \]

The Sarsa update is

\[ q_{t+1}(s_t,a_t) = q_t(s_t,a_t) + \alpha_t(s_t,a_t)\delta_t. \]

Equivalently,

\[ q_{t+1}(s_t,a_t) = q_t(s_t,a_t) - \alpha_t(s_t,a_t) \left[ q_t(s_t,a_t) - \left( r_{t+1} + \gamma q_t(s_{t+1},a_{t+1}) \right) \right]. \]

For all other state-action pairs,

\[ q_{t+1}(s,a) = q_t(s,a), \quad \forall (s,a)\neq(s_t,a_t). \]

Sarsa is threfore an on-policy TD method because the action used in the TD target is the actual next action sampled from the same policy that generates the trajectory.

Expected Sarsa: target is the policy expectation

Sarsa uses one sampled next action \(a_{t+1}\). Expected Sarsa instead uses the expected action value under policy \(\pi\).

If we know the full action distribution \(\pi(\cdot\mid s_{t+1})\), we can define the TD target as

\[ \bar q_t^{\text{Expected Sarsa}} = r_{t+1} + \gamma \sum_{a'} \pi(a'\mid s_{t+1})q_t(s_{t+1},a'). \]

Instead of using one sampled behavior action, Expected Sarsa averages over all possible next actions under the policy.

Expected Sarsa is still on-policy if the policy used in the expectation is the same policy that generates the data. Compared with Sarsa, it can reduce variance because it does not rely on a single sampled next action. The cost is that we need access to the action distribution and, in discrete action spaces, must sum over actions.

Q-learning: target action is greedy

Another possibility is to ignore the actual next behavior action and bootstrap from the action with the largest estimated Q-value.

Define

\[ a^* = \arg\max_{a'} q_t(s_{t+1},a'). \]

Then the Q-learning target is

\[ \bar q_t^{\text{Q-learning}} = r_{t+1} + \gamma q_t(s_{t+1},a^*). \]

Equivalently,

\[ \bar q_t^{\text{Q-learning}} = r_{t+1} + \gamma \max_{a'} q_t(s_{t+1},a'). \]

This gives Q-learning, which will be introduced in the next post.

The key difference from Sarsa is that Q-learning does not bootstrap from the next action actually selected by the behavior policy. Instead, it bootstraps from the greedy action according to the current Q-function.

Therefore, Q-learning is an off-policy TD method. The behavior policy may be exploratory, for example \(\epsilon\)-greedy, but the TD target corresponds to a greedy target policy.

Actor-critic style Q targets

In modern deep RL methods, especially with actor-critic methods, the target action may be produced by a learned actor neural network.

For example, in deterministic actor-critic methods, the TD target can be written as

\[ \bar q_t = r_{t+1} + \gamma Q_{\bar\psi} \left( s_{t+1}, \pi_{\bar\eta}(s_{t+1}) \right), \]

where \(Q_{\bar\psi}\) is a target critic and \(\pi_{\bar\eta}\) is a target actor.

Here the target action is

\[ a_{\text{target}} = \pi_{\bar\eta}(s_{t+1}). \]

This is also an action-value TD target. The behavior action in the data may have been generated by an exploratory behavior policy, but the bootstrap action in the TD target is produced by the actor. Therefore, this is typically an off-policy TD target.

This form appears in methods such as DDPG, TD3, and many deep model-based RL methods such as TD-MPC and DreamerV3.

TD, MC, and dynamic programming

TD learning lies between Monte Carlo learning and dynamic programming.

Monte Carlo learning uses sampled full returns:

\[ G_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots. \]

Dynamic programming uses the known transition model to compute exact expectations in the Bellman equation:

\[ v_\pi(s) = \sum_a \pi(a\mid s) \sum_{s',r} p(s',r\mid s,a) [ r+\gamma v_\pi(s') ]. \]

TD learning uses sampled transitions like MC, but bootstraps like dynamic programming:

\[ r_{t+1} + \gamma v_t(s_{t+1}). \]

Therefore:

• MC does not bootstrap but requires full returns.
• Dynamic programming bootstraps but requires a known model \(p(s′,r∣s,a)\).
• TD bootstraps from sampled transitions and does not require a known model.

Convergence of TD learning

For tabular policy evaluation, TD learning has strong convergence guarantees.

Theorem:

By the TD algorithm, \(v_t(s)\) converges with probability \(1\) to \(v_\pi(s)\) for all \(s\in\mathcal{S}\) as \(t\to\infty\), if

\[ \sum_t \alpha_t(s) = \infty \]

and

\[ \sum_t \alpha_t^2(s) < \infty \]

for all \(s\in\mathcal{S}\).

Remarks:

• The theorem applies to estimating the value function of a fixed policy \(\pi\).
• The condition \(\sum_t \alpha_t(s)=\infty\) ensures that learning does not stop too early.
• The condition \(\sum_t \alpha_t^2(s)<\infty\) ensures that the learning rate eventually becomes small enough for convergence.
• Each state must be visited sufficiently often.
• In practice, deep RL often uses constant learning rates, neural networks, replay buffers, and target networks. These settings are much more complex than the tabular convergence theorem.

TD-style learning in MBRL

TD learning is often introduced in tabular model-free RL, but its core idea—training value functions with bootstrapped targets—extends directly to modern deep RL and model-based RL, especially world-model-based RL.

In many world-model agents, the value network or Q-network is trained with targets of the form

\[ \text{finite-horizon rewards} + \text{bootstrapped terminal value}. \]

These methods are usually not called “TD learning algorithms” as a whole, but their value-learning components are TD-style learning in this broader sense.

Dreamer-style world models

Dreamer-style methods learn a latent world model and train actor-critic agents inside imagined trajectories. The critic is usually trained with bootstrapped returns such as \(\lambda\)-returns.

The general structure is

\[ \text{world model rollout} \rightarrow \text{imagined rewards} \rightarrow \lambda\text{-return target} \rightarrow \text{critic update}. \]

The actor is then optimized using the imagined trajectories and the learned value estimates.

Thus, although Dreamer is not called a TD algorithm in its title, its value learning uses TD-style bootstrapped targets.

DIAMOND

DIAMOND is a diffusion world model for Atari. Its RL agent is trained entirely in imagination. The policy is trained with REINFORCE with a value baseline, and the value network is trained using a Bellman error with \(\lambda\)-returns.

In DIAMOND’s notation, the world model predicts imagined observations \(x_t\), rewards \(r_t\), and termination indicators \(d_t\). The \(\lambda\)-return is defined recursively as

\[ \Lambda_t = r_t + \gamma(1-d_t) \left[ (1-\lambda)V_\phi(x_{t+1}) + \lambda \Lambda_{t+1} \right], \]

with terminal condition

\[ \Lambda_H = V_\phi(x_H). \]

The value network is trained by minimizing

\[ L_V(\phi) = \mathbb{E}_{\pi_\phi} \sum_{t=0}^{H-1} \left( V_\phi(x_t) - \operatorname{sg}(\Lambda_t) \right)^2. \]

This is a TD-style value objective. The rollout is truncated at horizon \(H\), and the future return after that horizon is approximated by the value network.

DIAMOND’s policy objective is REINFORCE with the value function as a baseline. Therefore, its RL part is not TD-MPC-style planning, but it still uses TD-style value learning.

TD-MPC and TD-MPC2

TD-MPC stands for Temporal Difference Model Predictive Control.

TD-MPC-style methods use a learned latent dynamics model for short-horizon model predictive control, and use a TD-learned terminal value function to estimate long-horizon return.

A typical planning objective has the form

\[ J(a_{0:H}) = \sum_{h=0}^{H-1} \gamma^h r(z_h,a_h) + \gamma^H V(z_H). \]

or, in Q-function form,

\[ J(a_{0:H}) = \sum_{h=0}^{H-1} \gamma^h r(z_h,a_h) + \gamma^H Q(z_H,a_H). \]

The first term is finite-horizon model rollout. The second term is terminal value bootstrap.

Thus, TD-MPC combines:

• a learned latent dynamics model;
• a learned reward model;
• a learned value or Q function;
• a policy prior;
• model predictive control at decision time.

The TD part is the value learning and terminal bootstrap. The MPC part is the online short-horizon trajectory optimization.