Introduction to Model Predictive Control

Posted on 2026-07-05 Edited on 2026-08-14 In AI Views:

Model Predictive Control (MPC) is a model-based planning method widely used in model-based reinforcement learning.

Sources:

Sutton and Barto. Reinforcement Learning: An Introduction.
Hansen et al. Temporal Difference Learning for Model Predictive Control.
Scannell et al. Discrete Codebook World Models for Continuous Control, ICLR 2025.

Introduction to Model Predictive Control

Its core idea is simple:

Use a model to plan several steps into the future, execute only the first action, and then replan at the next step.

MPC is also called receding-horizon control because the planning horizon keeps moving forward as the agent interacts with the environment.

The basic idea

Suppose the agent is at state \(s_t\). MPC considers a future action sequence:

\[ a_{t:t+H-1} = (a_t,a_{t+1},\dots,a_{t+H-1}). \]

Using a model, it predicts what will happen if this action sequence is executed:

\[ \hat{s}_{t+1} = f(\hat{s}_t,a_t), \]

\[ \hat{s}_{t+2} = f(\hat{s}_{t+1},a_{t+1}), \]

and so on.

Then MPC selects the action sequence with the highest predicted return:

\[ a^*_{t:t+H-1} = \arg\max_{a_{t:t+H-1}} J(a_{t:t+H-1},s_t). \]

However, MPC does not execute the whole sequence. It only executes the first action:

\[ a_t^*. \]

At the next environment step, it observes a new state and plans again.

The control loop is:

observe current state
plan H steps into the future
execute only the first action
observe the next state
replan

This is the key difference between MPC and open-loop planning. MPC continuously corrects itself using new observations.

Does MPC require a model?

Yes. MPC needs a predictive model because it must evaluate candidate future action sequences.

The model can be:

known physical model
simulator
learned dynamics model
learned latent world model

In classical control, the model may be given by physics.

In model-based RL, the model is often learned from data:

\[ \hat{s}_{t+1} = f_\theta(\hat{s}_t,a_t). \]

If the raw observation is high-dimensional, planning is often done in latent space:

\[ z_t = e_\theta(o_t), \]

\[ \hat{z}_{t+1} = f_\theta(\hat{z}_t,a_t). \]

Then the planner searches for actions in latent space instead of directly in pixel space.

MPC objective

A simple finite-horizon MPC objective is:

\[ J(a_{0:H-1},s_0) = \sum_{h=0}^{H-1} \gamma^h r(\hat{s}_h,a_h), \]

subject to

\[ \hat{s}_0=s_0, \]

\[ \hat{s}_{h+1}=f(\hat{s}_h,a_h). \]

This objective says:

simulate H steps
sum predicted rewards
choose the action sequence with the largest sum

In RL, we often care about long-horizon return. But MPC usually plans only a short horizon. Therefore, it is common to add a terminal value bootstrap:

\[ J(a_{0:H-1},s_0) = \sum_{h=0}^{H-1} \gamma^h r(\hat{s}_h,a_h) + \gamma^H V(\hat{s}_H). \]

If we use a Q-function instead of a state-value function, the terminal bootstrap can be:

\[ J(a_{0:H},s_0) = \sum_{h=0}^{H-1} \gamma^h r(\hat{s}_h,a_h) + \gamma^H Q(\hat{s}_H,a_H). \]

The first term evaluates the short-horizon plan. The terminal value estimates what happens after the planning horizon.

So the objective has the same structure as a TD target:

\[ \text{finite-horizon rewards} + \text{bootstrapped terminal value}. \]

Generic MPC pseudocode

A generic MPC loop is:

for each environment step t:

    observe current state s_t

    # solve a finite-horizon planning problem
    best_sequence = plan(
        initial_state=s_t,
        model=f,
        reward=r,
        horizon=H
    )

    # execute only the first action
    a_t = best_sequence[0]
    s_{t+1}, r_{t+1} = env.step(a_t)

    # discard the rest or use it to warm-start the next planning step

The planner can be implemented in different ways. The most common sampling-based methods are random shooting, CEM, and MPPI.

Why sample action sequences?

In sampling-based MPC, one sample is usually a whole future action sequence:

\[ a_{0:H-1} = (a_0,a_1,\dots,a_{H-1}). \]

If each action has dimension \(d_a\), then one action sequence has dimension:

\[ H d_a. \]

For example, if

\[ H=10,\quad d_a=4, \]

then one candidate action sequence is a \(40\)-dimensional vector.

This is why MPC in continuous action spaces often samples from a high-dimensional Gaussian distribution over action sequences:

\[ a_{0:H-1} \sim \mathcal{N} ( \mu_{0:H-1}, \operatorname{diag}(\sigma^2_{0:H-1}) ). \]

Here \(\mu_{0:H-1}\) and \(\sigma^2_{0:H-1}\) are sequences of means and variances, one for each future time step.

Common MPC solvers

Random shooting

Random shooting is the simplest sampling-based MPC solver.

It samples many action sequences, rolls each one out in the model, and chooses the best one.

def random_shooting_plan(s0, model, reward, H, num_samples):

    best_return = -float("inf")
    best_sequence = None

    for i in range(num_samples):

        actions = sample_random_action_sequence(H)

        s = s0
        score = 0

        for h in range(H):
            a = actions[h]
            score += gamma**h * reward(s, a)
            s = model(s, a)

        if score > best_return:
            best_return = score
            best_sequence = actions

    return best_sequence

Random shooting is simple, but it can be inefficient in high-dimensional action spaces because it does not improve its sampling distribution.

CEM

CEM, or Cross-Entropy Method, improves random shooting by maintaining a sampling distribution over action sequences.

Usually, this distribution is a diagonal Gaussian:

\[ a_{0:H-1} \sim \mathcal{N} ( \mu_{0:H-1}, \operatorname{diag}(\sigma^2_{0:H-1}) ). \]

The procedure is:

sample action sequences from current Gaussian
evaluate each sequence
keep top-K elite sequences
refit Gaussian to elites
repeat for several iterations
execute the first action of the final best sequence

Pseudocode:

def cem_plan(s0, model, reward, H, num_samples, num_elites, num_iters):

    mu = zeros([H, action_dim])
    sigma = ones([H, action_dim])

    for iteration in range(num_iters):

        candidates = sample_gaussian_sequences(mu, sigma, num_samples)

        scores = []

        for actions in candidates:
            score = rollout_score(s0, actions, model, reward)
            scores.append(score)

        elites = select_top_k(candidates, scores, k=num_elites)

        mu = mean(elites)
        sigma = std(elites)

    return mu

CEM uses hard elite selection. Only the best samples are used to update the distribution.

MPPI

MPPI, or Model Predictive Path Integral control, is another sampling-based trajectory optimization method.

Like CEM, MPPI samples action sequences and evaluates them. But instead of only using top elites, MPPI assigns a soft weight to each trajectory according to its return.

A typical weight has the form:

\[ w_i \propto \exp(\tau J_i), \]

where \(J_i\) is the predicted return of the \(i\)-th action sequence, and \(\tau\) is an inverse temperature.

Higher-return trajectories get larger weights.

Pseudocode:

def mppi_plan(s0, model, reward, H, num_samples, num_iters):

    mu = zeros([H, action_dim])
    sigma = ones([H, action_dim])

    for iteration in range(num_iters):

        candidates = sample_gaussian_sequences(mu, sigma, num_samples)

        scores = []

        for actions in candidates:
            score = rollout_score(s0, actions, model, reward)
            scores.append(score)

        weights = softmax(tau * scores)

        mu = weighted_mean(candidates, weights)
        sigma = weighted_std(candidates, weights)

    return mu

The difference is:

CEM:
  hard elite selection

MPPI:
  soft return-based weighting

Hybrid CEM-MPPI

Many practical methods use a hybrid between CEM and MPPI.

The idea is:

sample many trajectories
select top-K elites
compute return-based weights only over elites
update the Gaussian using weighted mean and variance

This is CEM-like because it selects elites, and MPPI-like because it uses return-based weights.

Pseudocode:

def hybrid_cem_mppi_plan(s0, model, reward, H, num_samples, num_elites, num_iters):

    mu = zeros([H, action_dim])
    sigma = ones([H, action_dim])

    for iteration in range(num_iters):

        candidates = sample_gaussian_sequences(mu, sigma, num_samples)

        scores = []

        for actions in candidates:
            score = rollout_score(s0, actions, model, reward)
            scores.append(score)

        elite_actions, elite_scores = select_top_k(candidates, scores, k=num_elites)

        weights = softmax(tau * elite_scores)

        mu = weighted_mean(elite_actions, weights)
        sigma = weighted_std(elite_actions, weights)

    return mu

This is close to what many modern learned-model MPC methods use.

Useful MPC tricks

Short horizon

MPC often uses a short planning horizon.

This is possible because MPC replans at every step. It does not need to solve the full episode at once.

A short horizon also reduces model error accumulation.

long rollout:
  more future information
  but more model error

short rollout:
  less model error
  but more myopic

Terminal value bootstrapping is often used to compensate for short horizons.

Terminal value bootstrap

If MPC plans only \(H\) steps, it still needs to estimate what happens after \(H\).

So we add a value term:

\[ J = \sum_{h=0}^{H-1} \gamma^h r(\hat{s}_h,a_h) + \gamma^H V(\hat{s}_H). \]

Or with a Q-function:

\[ J = \sum_{h=0}^{H-1} \gamma^h r(\hat{s}_h,a_h) + \gamma^H Q(\hat{s}_H,a_H). \]

This makes short-horizon planning closer to the full RL objective.

Warm start

MPC solves a new planning problem at every environment step. But consecutive planning problems are very similar.

Suppose the previous planned sequence was:

\[ (a_0,a_1,a_2,\dots,a_{H-1}). \]

MPC executes only \(a_0\). At the next step, the remaining actions

\[ (a_1,a_2,\dots,a_{H-1}) \]

are still useful.

So the next planning problem can be initialized with:

\[ (a_1,a_2,\dots,a_{H-1},a_{\text{new}}). \]

This is called warm start.

If the action sequence has shape

\[ [H,d_a], \]

then warm start shifts along the time dimension:

new_mu[0:H-1, :] = old_mu[1:H, :]
new_mu[H-1, :] = initial_last_action()

It does not shift along the action dimension.

Policy prior

MPC does not require a learned actor. Classical MPC often has no actor at all.

However, if we already have a learned policy, it can be used to propose candidate action sequences.

This is sometimes called a policy prior.

The word “prior” here does not mean a strict Bayesian prior. It means a reasonable proposal before planning.

random samples:
  explore many possible action sequences

policy-prior samples:
  provide action sequences that are likely to be reasonable

planner:
  evaluates all candidates and chooses the best

For example:

def sample_policy_prior_sequence(s0, policy, model, H):

    actions = []
    s = s0

    for h in range(H):
        a = policy(s)
        actions.append(a)
        s = model(s, a)

    return actions

The planner does not blindly trust the policy. It only treats policy-generated sequences as additional candidates.

Replanning

MPC is robust because it replans after every real environment step.

Even if the model prediction is imperfect, the agent observes the real next state and corrects the plan.

plan 10 steps
execute 1 step
observe real next state
plan again

This is why MPC can work with imperfect models, especially when the planning horizon is short.

MPC with learned world models

In modern model-based RL, MPC often uses a learned world model and a learned actor-critic.

A typical latent-space pipeline is:

observation
-> encoder
-> latent state
-> latent dynamics rollout
-> predicted rewards
-> terminal value/Q bootstrap
-> action-sequence optimization

For example:

\[ z_0=e_\theta(o), \]

\[ \hat{z}_{h+1}=f_\theta(\hat{z}_h,a_h), \]

\[ \hat{r}_h=R_\xi(\hat{z}_h,a_h). \]

The planner maximizes:

\[ J(a_{0:H},o) = \sum_{h=0}^{H-1} \gamma^h R_\xi(\hat{z}_h,a_h) + \gamma^H Q_\psi(\hat{z}_H,a_H). \]

This avoids planning directly in observation space.