Chapter 1
Principles and Theory of MuZero
What happens when we let a reinforcement learning agent master abstract planning, without ever seeing the rules of its environment? This chapter pulls back the curtain on MuZero's fascinating theoretical foundations, revealing how it elegantly bypasses the need for full environment models to achieve state-of-the-art results. By closely examining MuZero's formulation, learning objectives, and search integration, you'll see how it challenges conventions in RL, combines the best of model-based and model-free thinking, and reshapes our approach to intelligent sequential decision-making.
1.1 Introduction to Model-based Reinforcement Learning
Model-based reinforcement learning (MBRL) constitutes a fundamental paradigm within the broader reinforcement learning (RL) framework, distinguished primarily by its explicit construction and utilization of an internal model of the environment. Unlike model-free methods, which directly learn policies or value functions through trial-and-error interactions, model-based approaches leverage predictive models to facilitate planning and policy generation. Formally, an MBRL agent typically maintains a transition model T approximating the environment's state dynamics, and a reward model R estimating immediate rewards, enabling the simulation of future trajectories without exhaustive real-world exploration.
At its core, the transition model T :
