A Taxonomy of Model-Based RL Algorithms

• 1
• 2 - Augmented Agents): Weber et al, 2017
• 3 - Based RL with Model-Free Fine-Tuning): Nagabandi et al, 2017
• 4 - Based Value Expansion): Feinberg et al, 2018
• 5
• 6
• 8 - based Actor-Critic): Pan et al, 2020

Codebase

Other

• mbrl-lib - Meta: Library for Model Based RL
• DI-engine - OpenDILab: Decision AI Engine

Papers

Classic Model-Based RL Papers

• Exploring Model-based Planning with Policy Networks
• PILCO: A Model-Based and Data-Efficient Approach to Policy Search
• Learning Complex Neural Network Policies with Trajectory Optimization
• Learning Continuous Control Policies by Stochastic Value Gradients
• Value Prediction Network
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Recurrent World Models Facilitate Policy Evolution
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• When to Trust Your Model: Model-Based Policy Optimization
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Model-Ensemble Trust-Region Policy Optimization
• Dream to Control: Learning Behaviors by Latent Imagination
• Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
• Dyna, an integrated architecture for learning, planning, and reacting

ICLR 2021

• RL Unplugged(RLU) - berkeley/d4rl)
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Control-Aware Representations for Model-based Reinforcement Learning
• Mastering Atari with Discrete World Models
• Model-Based Visual Planning with Self-Supervised Functional Distances
• Model-Based Offline Planning
• Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation
• On the role of planning in model-based deep reinforcement learning
• Representation Balancing Offline Model-based Reinforcement Learning
• Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

ICLR 2022

• Revisiting Design Choices in Offline Model Based Reinforcement Learning
• Value Gradient weighted Model-Based Reinforcement Learning
• Planning in Stochastic Environments with a Learned Model
• Policy improvement by planning with Gumbel
• Model-Based Offline Meta-Reinforcement Learning with Regularization
• On-Policy Model Errors in Reinforcement Learning
• A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning
• Information Prioritization through Empowerment in Visual Model-based RL
• Transfer RL across Observation Feature Spaces via Model-Based Regularization
• Learning State Representations via Retracing in Reinforcement Learning
• Model-augmented Prioritized Experience Replay
• Evaluating Model-Based Planning and Planner Amortization for Continuous Control
• Gradient Information Matters in Policy Optimization by Back-propagating through Model
• Pareto Policy Pool for Model-based Offline Reinforcement Learning
• Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage
• Know Thyself: Transferable Visual Control Policies Through Robot-Awareness
• pybullet

ICLR 2023

• Transformers are Sample-Efficient World Models
• Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
• User-Interactive Offline Reinforcement Learning
• CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning
• Efficient Offline Policy Optimization with a Learned Model
• Efficient Planning in a Compact Latent Action Space
• Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function
• MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
• Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objective
• The Benefits of Model-Based Generalization in Reinforcement Learning
• Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning
• Planning Goals for Exploration
• Making Better Decision by Directly Planning in Continuous Control
• Latent Variable Representation for Reinforcement Learning
• SpeedyZero: Mastering Atari with Limited Data and Time
• Transformer-based World Models Are Happy With 100k Interactions
• On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning
• Become a Proficient Player with Limited Data through Watching Pure Videos
• EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model
• Choreographer: Learning and Adapting Skills in Imagination
• URLB benchmark
• adroit - world](https://github.com/rlworkgroup/metaworld), deepmind control suite
• brax
• atari dataset

ICLR 2024

• TD-MPC2: Scalable, Robust World Models for Continuous Control
• Robust Model Based Reinforcement Learning Using L1 Adaptive Control
• Mastering Memory Tasks with World Models
• Learning Hierarchical World Models with Adaptive Temporal Abstractions from Discrete Latent Dynamics
• COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL
• Efficient Multi-agent Reinforcement Learning by Planning
• Differentiable Trajectory Optimization as a Policy Class for Reinforcement and Imitation Learning
• DMBP: Diffusion model based predictor for robust offline reinforcement learning against state observation perturbations
• MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning
• Point Robot Navigation, Escape Room
• Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
• DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing
• Informed POMDP: Leveraging Additional Information in Model-Based RL
• varying mountain hike
• Privileged Sensing Scaffolds Reinforcement Learning
• Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
• NuScenes
• Policy Rehearsing: Training Generalizable Policies for Reinforcement Learning
• Efficient Dynamics Modeling in Interactive Environments with Koopman Theory
• Combining Spatial and Temporal Abstraction in Planning for Better Generalization
• bsuite - maze)
• gymnasium robotics
• MiniHack
• smac
• robodesk - dynamics/)
• d4rl

ICLR 2025

• Learning Transformer-based World Models with Contrastive Predictive Coding
• Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
• OptionZero: Planning with Learned Options
• MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL
• Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks
• Learning to Search from Demonstration Sequences
• Open-World Reinforcement Learning over Long Short-Term Imagination
• MaestroMotif: Skill Design from Artificial Intelligence Feedback
• Geometry-aware RL for Manipulation of Varying Shapes and Deformable Objects
• M^3PC: Test-time Model Predictive Control using Pretrained Masked Trajectory Model
• Offline Model-Based Optimization by Learning to Rank
• Monte Carlo Planning with Large Language Model for Text-Based Games
• Interpreting Emergent Planning in Model-Free Reinforcement Learning
• Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient
• Zero-shot Model-based Reinforcement Learning using Large Language Models
• On Rollouts in Model-Based Reinforcement Learning
• Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning
• Discrete Codebook World Models for Continuous Control

ICLR 2026

• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning
• Differentiable Model Predictive Control on the GPU
• Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling
• World-In-World: World Models in a Closed-Loop World
• R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation
• Learning Massively Multitask World Models for Continuous Control
• Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamics
• WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Control
• Object-Centric World Models from Few-Shot Annotations for Sample-Efficient Reinforcement Learning
• Horizon Imagination: Efficient On-Policy Rollout in Diffusion World Models
• Sparse Imagination for Efficient Visual World Model Planning
• MOBODY: Model-Based Off-Dynamics Offline Reinforcement Learning
• Scalable Offline Model-Based RL with Action Chunks
• Code World Models for General Game Playing
• One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planning
• Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning
• Regret-Guided Search Control for Efficient Learning in AlphaZero
• WMPO: World Model-based Policy Optimization for Vision-Language-Action Models
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation
• Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
• Vid2World: Crafting Video Diffusion Models to Interactive World Models
• Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
• WorldGym: World Model as An Environment for Policy Evaluation
• One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration
• ExoPredicator: Learning Abstract Models of Dynamic Worlds for Robot Planning
• Learning Koopman Representations with Controllability Guarantees
• Model Predictive Adversarial Imitation Learning for Planning from Observation
• Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data

ICML 2021

• sparse metaworld tasks
• Conservative Objective Models for Effective Offline Model-Based Optimization
• Continuous-Time Model-Based Reinforcement Learning
• Model-Based Reinforcement Learning via Latent-Space Collocation
• Model-Free and Model-Based Policy Evaluation when Causality is Uncertain
• Muesli: Combining Improvements in Policy Optimization
• Vector Quantized Models for Planning
• chess datasets
• PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Exploration
• Temporal Predictive Coding For Model-Based Planning In Latent Space
• Model-based Reinforcement Learning for Continuous Control with Posterior Sampling
• A Sharp Analysis of Model-based Reinforcement Learning with Self-Play
• pendulum, cartPole and acrobot
• design-bench
• ope-tools
• sparse metaworld tasks
• Model-Free and Model-Based Policy Evaluation when Causality is Uncertain
• Vector Quantized Models for Planning

ICML 2022

• DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
• Denoised MDPs: Learning World Models Better Than the World Itself
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models and Amortized Policy Search
• Towards Adaptive Model-Based Reinforcement Learning
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation
• Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
• Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization
• Design-Bench Benchmark Tasks
• GridWorldLoCA, ReacherLoCA, MountaincarLoCA
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation
• SMART
• Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization
• Temporal Difference Learning for Model Predictive Control
• Denoised MDPs: Learning World Models Better Than the World Itself
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models and Amortized Policy Search
• Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
• Towards Adaptive Model-Based Reinforcement Learning

ICML 2023

• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels
• Reparameterized Policy Learning for Multimodal Trajectory Optimization
• Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy
• Predictable MDP Abstraction for Unsupervised Model-Based RL
• Investigating the Role of Model-Based Learning in Exploration and Transfer
• The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms
• Helicopter, WideTree, Linear Dynamical System, Maze - py)
• The Benefits of Model-Based Generalization in Reinforcement Learning
• ProcMaze, ButtonGrid, PanFlute
• STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning
• DeepSea
• Model-based Reinforcement Learning with Scalable Composite Policy Gradient Estimators
• Reinforcement Learning with History Dependent Dynamic Contexts
• MovieLens dataset
• Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning
• Simplified Temporal Consistency Reinforcement Learning
• Curious Replay for Model-based Adaptation
• On Many-Actions Policy Gradient
• Posterior Sampling for Deep Reinforcement Learning