{ "id": "world-model-ai", "title": "World model (AI)", "lang": "en", "source_lang": "en", "summary": "An internal model of an environment that enables an AI system to predict events and anticipate the consequences of actions.", "category": "computer-science", "subcategory": "artificial-intelligence", "type": "concept", "aliases": [ "AI world model", "artificial intelligence world model" ], "generated_at": "2026-10-14", "model": "gpt-6.1-sol", "url": "https://aiwiki.page/en/world-model-ai/", "translations": { "zh-Hans": { "url": "https://aiwiki.page/zh/world-model-ai/", "json": "https://aiwiki.page/zh/world-model-ai.json" } }, "sections": [ { "heading": "", "text": "A world model in artificial intelligence is an internal representation of an environment and its dynamics, used to predict how situations evolve and, in action-conditioned systems, what may happen when an agent acts. Learned world models can represent observations directly or encode them in a compact latent space. Some generate future images or other sensory observations; others predict only abstract features, rewards, or quantities needed for decision-making. The term therefore describes a functional role rather than a single model architecture. (proceedings.mlr.press)\n\nWorld models are important in model-based reinforcement learning because they allow agents to plan or learn from simulated experience rather than relying exclusively on interaction with an external environment. Research also encompasses interactive environment generation and visual representations that support physical prediction and robotic planning. These approaches differ in what they model, how they learn, and how their predictions are used; their usefulness depends not just on predictive fidelity, but on whether imagined outcomes support reliable decisions beyond the training experience. (nature.com)" }, { "heading": "Definition and conceptual boundaries", "text": "An environment model represents aspects of an agent’s surroundings and, in its predictive form, specifies how those surroundings evolve. In artificial intelligence, world model commonly denotes a learned predictive model of this kind, sometimes including the mechanisms that encode observations into an internal state. A representation of the present and a model of change perform different functions: encoding a camera image describes what is currently visible, whereas predicting a subsequent observation or internal state describes temporal evolution. David Ha and Jürgen Schmidhuber’s 2018 World Models system makes this distinction explicit: its visual component compresses individual frames, while its recurrent component predicts future compressed representations using previous information and the agent’s action. “World” can therefore mean a particular game or control environment, rather than a comprehensive model of physical reality. (worldmodels.github.io)\n\nThe term encompasses several overlapping uses. In reinforcement learning and control theory, it often refers to learned dynamics used to anticipate the consequences of candidate actions or generate experience for learning a controller. In generative AI, it can describe a model that produces an interactive environment: Jake Bruce and colleagues’ 2024 Genie generates visual sequences that users can influence frame by frame, despite its training videos lacking recorded action labels. In predictive representation learning, the emphasis can instead be on anticipating features relevant to physical events without synthesizing their full visual appearance. The 2025 V-JEPA 2 work illustrates this distinction by separating action-free video pretraining from subsequent action-conditioned modeling for robotic planning. These usages share temporal prediction but differ in their outputs and access to actions. (worldmodels.github.io)\n\nA world model is not synonymous with an entire agent. A policy selects actions; a value function estimates expected cumulative reward; a dynamics model predicts what follows from a situation and, where applicable, an action. These functions may share representations, but they answer different questions. Memory retains information from earlier observations and actions; it can support prediction without being equivalent to a predictive model. In Ha and Schmidhuber’s system, recurrent memory belongs to the predictive component, while a separately trained controller chooses actions. MuZero, introduced in 2019, further demonstrates that a useful learned model need not reconstruct future images: its internal dynamics support predictions of reward, policy, and value for planning. Neither a standalone model nor its internal state is the external environment that ultimately supplies observations and outcomes. (worldmodels.github.io)\n\nA supplied simulator and a learned world model can both provide predicted transitions, but their construction differs. A supplied simulator exposes dynamics already implemented for the planning system; a learned model estimates predictive relationships from data. Model-based decision-making does not inherently require learning the simulator. Conversely, a learned model can function as a surrogate simulator without reproducing every environmental detail. A static knowledge base records facts or relations, but storing descriptions of objects or events does not by itself provide a mechanism for advancing a situation through time. For the predictive usage of world model, the relevant capability is producing subsequent states, observations, or task-relevant quantities—not simply retaining descriptive information. (pubsonline.informs.org)\n\nThe boundary with ordinary video generation depends especially on action responsiveness. A generator may produce temporally plausible motion without accepting a sequence of control inputs whose effects can be tested. An interactive model must additionally respond to actions as the sequence unfolds, as Genie’s frame-by-frame control illustrates. Action conditioning is not mandatory for every broader use of world model: observational predictive models can learn temporal regularities without it. It is, however, a crucial distinction when a model is intended to compare alternative actions or act as a controllable environment; visual continuity alone does not establish that capability. (arxiv.org)" }, { "heading": "Historical development", "text": "Richard Sutton’s Dyna architecture, described in publications in 1990 and 1991, integrated learning an environment model with learning how to act. Rather than treating planning and reinforcement learning as separate procedures, Dyna allowed both to update an agent’s decision-making machinery. Experience obtained through interaction trained a model of the consequences of actions; the model then supplied simulated experience for additional learning updates between real interactions. Sutton’s 1990 experiments included a navigation task in which an agent simultaneously learned through trial and error, revised its world model, and used the evolving model to plan routes. This established a concrete mechanism for reusing experience through model-generated transitions, decades before the image-based neural systems subsequently associated with the term “world model.” (mlanthology.org)\n\nDavid Ha and Jürgen Schmidhuber’s World Models, first released in March 2018, investigated compressed spatial and temporal representations of visual environments. A variational autoencoder compressed individual images, while a recurrent neural network learned to predict subsequent representations conditioned on actions. A small controller used the compressed representation and recurrent memory to select actions. In their CarRacing experiment, training began with 10,000 randomly generated episodes, and each image was compressed into a 32-dimensional vector. Their VizDoom experiment demonstrated a different use of the model: the controller was optimized entirely inside a learned latent environment and then transferred to the original game environment. The environment model itself was learned from recordings collected through random interaction; training the controller “inside” the model did not eliminate the need for environmental data. (arxiv.org)\n\nPlaNet, developed by Danijar Hafner and colleagues, appeared as a preprint in November 2018 and was published at the International Conference on Machine Learning in June 2019. It addressed online planning directly from pixel observations: instead of choosing actions by simulating full rendered images, it evaluated candidate action sequences in a learned latent space. Its recurrent state-space model combined deterministic memory with stochastic latent transitions, allowing the representation to retain temporal information while modelling uncertain outcomes. The experiments covered visual continuous-control tasks involving contact dynamics, partial observability, and sparse rewards. Historically, PlaNet connected learned visual representations to repeated, action-conditioned planning, rather than using the model only as a source of features for a controller. (arxiv.org)\n\nLate 2019 brought distinct approaches to the information a model should retain and how an agent should use it. MuZero, introduced by Julian Schrittwieser and colleagues, combined tree search with learned dynamics that predicted rewards, action-selection policies, and values, without requiring reconstruction of future observations. Its evaluations included Atari games, Go, chess, and shogi. Dreamer, introduced by Hafner and colleagues in a December 2019 preprint and presented at ICLR 2020, instead trained action and value models through imagined latent trajectories. Unlike PlaNet’s online search over action sequences, Dreamer used imagination to learn a policy that could subsequently select actions directly. Later Dreamer versions extended this approach across more diverse control settings; DreamerV3 was introduced in 2023 and published in an expanded journal study in 2025. (arxiv.org)\n\nGenie, introduced by Jake Bruce and colleagues in February 2024, extended this research toward interactive environments learned from observational video rather than recorded action-labelled trajectories. Its 11-billion-parameter model was trained on more than 200,000 hours of publicly available gaming video without action or text annotations. A learned latent-action mechanism enabled frame-by-frame control of generated environments, including environments initialized from previously unseen images. This made interactive environment generation itself a research target, alongside the earlier emphasis on models used internally for planning or policy learning. (arxiv.org)" }, { "heading": "Mathematical formulation and partial observability", "text": "At discrete time \\(t\\), let \\(s_t\\) denote the environment’s state, \\(o_t\\) the agent’s observation, and \\(a_t\\) its action. A transition distribution \\(p(s_{t+1}\\mid s_t,a_t)\\) describes the possible next states. The Markov property assumes that the current state and action contain the information needed for this prediction, without separately conditioning on earlier history. A deterministic transition instead specifies \\(s_{t+1}=f(s_t,a_t)\\); it is the special case in which the transition distribution concentrates on one outcome. A probabilistic model can represent several possible futures rather than returning only a single prediction. (cassandra.org)\n\nAn observation model \\(p(o_t\\mid s_t)\\) describes how hidden states produce sensor readings. Observations need not identify states uniquely: one image can show an object’s position without revealing its velocity, while occlusion can hide objects entirely. This makes visual control a partially observable Markov decision process rather than necessarily a fully observed Markov decision process. In Danijar Hafner and colleagues’ 2019 PlaNet experiments, observations were \\(64\\times64\\times3\\)-pixel images; in the cartpole task, the fixed camera allowed the cart to move out of sight. Predicting its subsequent position therefore required information beyond the current image. (proceedings.mlr.press)\n\nA formal way to retain that information is a belief state, the conditional distribution \\(b_t(s)=p(s_t=s\\mid o_{1:t},a_{1:t-1})\\). Under the transition and observation assumptions above, Bayesian filtering updates it through\n\n\\[b_{t+1}(s') \\propto p(o_{t+1}\\mid s') \\int p(s'\\mid s,a_t)b_t(s)\\,ds,\\]\n\nwith a sum replacing the integral for discrete states. The transition first predicts where the system might move; the new observation then reweights those possibilities. An exact belief summarizes the history for subsequent decisions, although computing and storing it can be impractical in large state spaces. (cassandra.org)\n\nLearned models commonly replace exact filtering with an inferred state in a latent space. Writing \\(z_t\\) for this representation distinguishes it from the physical state \\(s_t\\). An inference model \\(q_\\phi(z_t\\mid o_{1:t},a_{1:t-1})\\) incorporates observed evidence, whereas a dynamics model \\(p_\\theta(z_{t+1}\\mid z_t,a_t)\\) predicts without the next observation. A recurrent neural network can compress history into memory \\(h_t\\), yielding an inference distribution \\(q_\\phi(z_t\\mid h_t,o_t)\\). PlaNet combines deterministic recurrent memory with a stochastic latent variable; the combined state supports prediction when an individual frame is insufficient. Such a learned summary approximates, rather than guarantees, the information retained by an exact belief. (proceedings.mlr.press)\n\nFor a fixed candidate action sequence and a Markov latent state, an open-loop rollout has the factorization\n\n\\[p_\\theta(z_{t+1:t+H}\\mid z_t,a_{t:t+H-1}) = \\prod_{k=t}^{t+H-1} p_\\theta(z_{k+1}\\mid z_k,a_k).\\]\n\nIf recurrent memory is used, \\(z_k\\) here denotes the complete predictive state, including that memory. An uncertain starting state is handled by integrating over its inferred distribution. Observation predictions can be added through factors \\(p_\\theta(o_{k+1}\\mid z_{k+1})\\), but decoding images is unnecessary when decisions use latent predictions directly. PlaNet, for example, evaluates predicted rewards in latent space without generating future images. (proceedings.mlr.press)\n\nControl-oriented models may additionally predict rewards and episode continuation. With a transition-based indexing convention, these can be written \\(p_\\theta(r_{t+1}\\mid z_t,a_t,z_{t+1})\\) and \\(p_\\theta(c_{t+1}\\mid z_{t+1})\\), where \\(c_{t+1}\\in\\{0,1\\}\\) indicates whether the episode continues. DreamerV3 predicts rewards and continuation from its combined recurrent and stochastic state; continuation weights prevent estimated returns from counting rewards after termination. Reward-based control needs a way to evaluate outcomes, but that quantity may be supplied externally rather than learned. Likewise, known termination rules can replace a continuation predictor. A value function may estimate returns beyond a finite rollout horizon, but is distinct from the transition model itself. (arxiv.org)" }, { "heading": "Representations and architectural families", "text": "World-model architectures differ in what they predict and which environmental details their internal states must preserve. Observation-space models predict future sensory observations, such as images, whereas compressed latent-state models advance an internal representation and optionally decode it into observations. A decoder can therefore support visual reconstruction without being required at every predicted step: PlaNet, published in 2019 by Danijar Hafner and colleagues, learns an observation model but performs planning entirely through latent transitions and reward predictions. Task-relevant architectures go further by omitting observation reconstruction and retaining information needed for specified decision-making outputs. MuZero exemplifies this approach through reward, policy, and value prediction. These distinctions concern predictive targets rather than mutually exclusive neural-network designs. (proceedings.mlr.press)\n\nAn encoder–decoder architecture separates sensory compression from reconstruction. In David Ha and Jürgen Schmidhuber’s 2018 World Models system, a variational autoencoder provides the visual component: its encoder maps an image to a distribution over latent vectors, and its decoder reconstructs images from sampled vectors. The CarRacing experiment compresses each \\(64 \\times 64\\)-pixel colour image into a 32-dimensional vector \\(z_t\\). A separate memory component combines a recurrent neural network with a mixture-density output layer, modelling \\(p(z_{t+1}\\mid z_t,a_t,h_t)\\), where \\(a_t\\) is the action and \\(h_t\\) is recurrent memory. Rather than outputting a single next code, it produces parameters of a Gaussian mixture from which possible successor codes can be sampled. Spatial compression and temporal prediction are thus separate learned operations; the controller receives the current code and recurrent memory, not a reconstructed image. (worldmodels.github.io)\n\nA recurrent state-space model integrates deterministic memory with stochastic latent dynamics. PlaNet’s architecture uses a deterministic recurrent state \\(h_t\\) alongside a stochastic state \\(z_t\\), with transitions expressible as \\(h_t=f(h_{t-1},z_{t-1},a_{t-1})\\) and \\(z_t\\sim p(z_t\\mid h_t)\\). The recurrent pathway carries information across time without requiring it to pass exclusively through successive random samples; the stochastic component represents distributions over latent states. When observations are available, an inference network estimates \\(q(z_t\\mid h_t,o_t)\\); during prediction without new observations, the model uses its transition distribution instead. Observation and reward decoders read the combined state. This differs from merely appending random noise to a deterministic image predictor: both memory and the inferred stochastic state participate in the model’s temporal representation. (proceedings.mlr.press)\n\nVideo-generative and joint-embedding predictive architectures differ principally in whether their predicted representations must support sensory reconstruction. Genie, introduced in 2024, tokenizes video into discrete codes and uses action-conditioned, temporally autoregressive dynamics to generate successive frames through a decoder. Its dynamics model predicts tokens within each frame using a masked-token procedure, rather than necessarily generating every spatial token sequentially. By contrast, V-JEPA uses self-supervised learning to predict encoded representations of masked video regions without pixel reconstruction. V-JEPA 2, introduced in 2025, subsequently adds action-conditioned prediction of future representations. Such feature predictions do not themselves specify rendered images. Autoregression and joint-embedding prediction are not opposites: a feature predictor can also be rolled forward autoregressively; the important distinction is the prediction space and its relationship to sensory detail. (arxiv.org)\n\nMuZero, introduced by Julian Schrittwieser and colleagues in 2019, uses three coupled functions: a representation function maps observation history into an internal state; a dynamics function maps that state and an action to a successor state and predicted reward; and a prediction function produces an action-selection policy and value. Repeated application supplies the quantities used by tree search without generating future observations. Its hidden states are not required to correspond to identifiable board configurations or reconstructed game screens. Unlike a generative latent model, MuZero is trained to preserve information useful for these planning outputs, not to reproduce the environment’s full observable appearance. (arxiv.org)" }, { "heading": "Training data and learning objectives", "text": "World-model training data can consist of trajectories collected by an interacting agent, stored experience sampled repeatedly during training, human or robot demonstrations, or observational video. Interaction trajectories typically record observations, actions, rewards, and episode boundaries; demonstrations can supply observation–action sequences without requiring the learner to discover those behaviours itself. Replaying stored sequences separates model updates from new environment interactions. In Danijar Hafner and colleagues’ 2019 PlaNet system, training began with random-action episodes and alternated between sampling stored sequence chunks and collecting additional experience. Video-only datasets provide temporal observations but not necessarily the control inputs responsible for transitions: Genie’s 2024 experiments, for example, treated robot demonstrations as videos and deliberately omitted their recorded actions. (proceedings.mlr.press)\n\nDifferent loss functions determine which information the learned state must preserve. Reconstruction losses encourage representations to retain information needed to reproduce observations; negative log-likelihood trains a specified predictive distribution to assign probability to recorded outcomes. PlaNet used fixed-variance Gaussian observation and reward models, making their negative log-likelihoods equivalent to squared error up to scaling and constants. Reward prediction retains information relevant to task outcomes, whereas continuation prediction retains information about whether an episode remains active. DreamerV3 combines observation reconstruction, reward prediction, and continuation prediction. Models trained to predict latent features instead compare predicted representations with encoded targets, without necessarily learning to reconstruct pixels; V-JEPA 2-AC uses an \\(L_1\\) feature-prediction loss. (proceedings.mlr.press)\n\nIn latent dynamics models, variational inference connects states inferred from observations with distributions predicted by the dynamics. With recurrent history \\(h_t\\), an encoder produces an approximate posterior \\(q_\\phi(z_t\\mid h_t,o_t)\\), while the dynamics predicts a prior \\(p_\\theta(z_t\\mid h_t)\\) without seeing the current observation. An evidence lower bound combines expected reconstruction log-likelihood with a Kullback–Leibler divergence penalty between these distributions. Reconstruction rewards informative states; the divergence penalizes observation-dependent information that the dynamics cannot predict. DreamerV3 separates this comparison into dynamics and representation losses using stop-gradient operators: one updates the predictor toward inferred states, while the other makes inferred representations more predictable. Its divergence losses are clipped below one nat, disabling their gradients beneath that threshold. (proceedings.mlr.press)\n\nTraining on sequences does not automatically mean directly training open-loop, multi-step predictions. An encoder may incorporate each new observation before the next transition is assessed. PlaNet introduced latent overshooting: the transition model is rolled forward across several steps using recorded actions, and its predicted state distributions are compared with observation-informed posteriors at corresponding times. This adds longer-range divergence penalties without decoding an image at every predicted step, unlike observation-space overshooting. The paper reported benefits for several dynamics models, but its final recurrent state-space agent did not require overshooting; the objective was an additional training method rather than a necessary component of every PlaNet implementation. (proceedings.mlr.press)\n\nObservational pretraining followed by action-conditioned adaptation separates learning visual regularities from learning controllable transitions. In 2025, V-JEPA 2 was pretrained through self-supervised learning on video and image data containing more than one million hours of internet video. Its action-conditioned version, V-JEPA 2-AC, then trained a predictor over a frozen encoder using fewer than 62 hours of DROID robot video. Here “unlabeled” meant that task identities, rewards, and success annotations were unused—not that control-related information was absent. Actions were constructed from changes in recorded seven-dimensional end-effector states. Training combined next-step feature prediction with a two-step autoregressive rollout loss. (arxiv.org)\n\nGenie instead learns a control representation when recorded inputs are unavailable. Its latent-action model encodes frame-to-frame transitions into a small discrete codebook and learns those codes through a next-frame reconstruction task. The dynamics model subsequently predicts video tokens conditioned on inferred codes and preceding frames. The 2024 platformer experiments used eight latent-action codes and approximately 30,000 hours of video. During generation, selecting a code supplies an intervention to the learned dynamics, although that code is not a supervised label for an original controller button. This differs from action-conditioned adaptation using measured robot commands or state changes: the control vocabulary itself is inferred from visual transitions. (arxiv.org)" }, { "heading": "Planning and learning through imagined experience", "text": "Online trajectory planning uses a learned model to evaluate actions before executing them. From the state inferred from recent observations, a planner proposes action sequences, predicts their resulting states and rewards, and selects a sequence with high predicted return. Under model-predictive control, or receding-horizon control, only the first action is executed; the next observation updates the state estimate, and planning starts again. This feedback distinguishes the method from executing an entire predicted sequence without further observation. PlaNet, published by Danijar Hafner and colleagues in 2019, implements this procedure in latent space. Its cross-entropy optimizer repeatedly samples candidate sequences and refits its sampling distribution to high-reward candidates. Although image reconstruction helps train the model, planning uses latent transitions and reward predictions rather than decoded images. PlaNet does not require a separate policy or value network to choose actions. (proceedings.mlr.press)\n\nTree search organizes imagined experience into branching alternatives rather than independently evaluating complete sequences. MuZero, introduced by Julian Schrittwieser and colleagues in 2019 and published in Nature in 2020, performs Monte Carlo tree search over learned hidden-state transitions. Its prediction network supplies action priors and state values; its dynamics network advances a hidden state under a proposed action and predicts the immediate reward. Search selects branches, expands new nodes, and backs up predicted rewards and values to update action estimates. Root visit counts then determine action selection and provide policy-training targets. Consequently, executing the network’s policy directly is not equivalent to running MuZero’s search: search performs additional model-based computation before acting. The original experiments used 800 simulations per search for Go, chess, and shogi, and 50 for Atari. (arxiv.org)\n\nDreamer instead uses imagined trajectories primarily to train a reusable actor–critic controller. Introduced by Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi in December 2019 and presented at ICLR 2020, the original algorithm separates three roles. The world model predicts latent transitions and rewards; the actor maps latent states to actions; the critic estimates the return obtainable from those states. Starting from states inferred from recorded experience, the actor supplies actions to the model, producing trajectories without further environmental interaction. The critic learns targets combining predicted rewards with bootstrapped value estimates, allowing finite imagined trajectories to support longer-horizon behavior. Actor optimization propagates analytic gradients through predicted transitions and rewards to increase estimated returns, while world-model parameters remain fixed during behavior updates. During environmental interaction, the actor selects actions from the inferred latent state without performing online trajectory search. (arxiv.org)\n\nModel-generated transitions can also augment ordinary experience replay. Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine’s 2019 model-based policy optimization (MBPO) fits a probabilistic ensemble to real transitions, samples starting states from the environmental replay buffer, and generates short branches using the current policy. Synthetic transitions enter a separate model buffer and are mixed with real experience for soft actor–critic updates. This separates the model rollout horizon from the task’s episode length: long-term value learning need not require equally long model simulations. In the reported Hopper experiments, a rollout schedule increasing from one to fifteen steps performed best, while fixed one-step rollouts retained much of the benefit. The model supplies training examples rather than an online search procedure. (papers.neurips.cc)\n\nInteraction efficiency and computational expense therefore measure different resources. MBPO matched the reported Ant performance of soft actor–critic using approximately 300,000 rather than three million environment steps, but employed 20–40 policy-gradient updates per environmental sample, alongside model fitting and synthetic-data generation. These step counts do not establish an equivalent reduction in elapsed training time. Online planners additionally incur repeated candidate-rollout or tree-search costs at decision time, whereas Dreamer places imagination-based behavior optimization in training and executes its learned actor during interaction. Reduced environmental sampling can thus coexist with substantial model-training, rollout-generation, and search computation. (papers.neurips.cc)" }, { "heading": "Uncertainty and model reliability", "text": "Aleatoric uncertainty concerns stochastic outcomes: even with known dynamics, the same state and action can produce different successors. Epistemic uncertainty concerns insufficient knowledge of those dynamics, particularly where training data are sparse; additional observations can reduce it. The distinction matters for exploration: repeatedly observing inherently random outcomes need not resolve uncertainty about the model. A predictive distribution can represent both sources, but its total variance alone does not distinguish them. (arxiv.org)\n\nKurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine’s 2018 probabilistic ensembles with trajectory sampling (PETS) used ensembles of bootstrapped neural networks. Each network predicted a Gaussian successor-state distribution: within-network variance represented aleatoric uncertainty, while disagreement between networks approximated epistemic uncertainty. Candidate action sequences were evaluated by propagating sampled state particles and averaging their cumulative rewards. In the \\(TS_1\\) variant, particles resampled their network at every step; in \\(TS_\\infty\\), each retained one network throughout the trajectory, preserving a consistent dynamics hypothesis. The experiments used five networks and 20 particles. This propagates uncertainty through successive predictions rather than replacing every possible successor with its mean. (arxiv.org)\n\nLong rollouts introduce a separate reliability problem: each predicted state becomes an input to the next prediction, so small errors can compound and move trajectories into regions where the model is less accurate. Policy improvement also changes which state–action pairs are visited, creating distribution shift relative to previously collected experience. Model exploitation occurs when optimization selects behavior because inaccurate dynamics predict unusually favorable returns, rather than because that behavior earns comparable rewards in the environment. This problem is particularly consequential in offline learning, where additional interaction cannot correct the misleading predictions. It is not necessary for the model to be inaccurate everywhere: an optimizer can preferentially select its advantageous errors. (papers.nips.cc)\n\nBounded rollout horizons limit exposure to compounding error. Michael Janner and colleagues’ 2019 model-based policy optimization (MBPO) branched short synthetic rollouts from states in real experience, separating the model’s prediction horizon from the task’s duration. Its online procedure repeatedly collected environmental transitions and retrained the model, refreshing coverage as the policy changed. In the reported Hopper experiment, predicted and environmental policy returns were highly correlated; even single-step synthetic rollouts provided an effective baseline. These findings support restricted model use, not universal reliability: the theoretical justification depends on assumptions or estimates concerning model error, policy change, and generalization beyond the training distribution. A short rollout can still encounter an inaccurate transition. (papers.nips.cc)\n\nUncertainty-aware methods can also modify the optimization objective. Tianhe Yu and colleagues’ 2020 model-based offline policy optimization (MOPO) penalized rewards according to estimated dynamics uncertainty, using\n\n\\[\\widetilde r(s,a)=r(s,a)-\\lambda u(s,a),\\]\n\nwhere \\(u(s,a)\\) estimates model error and \\(\\lambda\\) controls the penalty. Its theoretical lower bound on environmental return requires an admissible estimator that upper-bounds transition error in the specified distributional distance, together with assumptions on the value-function class. The implementation instead used a heuristic based on predicted model variance. Consequently, the theorem does not establish that an arbitrary ensemble’s variance provides a valid error bound. Unlike online data refresh, this penalty operates with a fixed dataset, trading potential gains outside its coverage against estimated prediction risk. (proceedings.neurips.cc)" }, { "heading": "Video-trained and interactive world models", "text": "Observational video supplies examples of object motion, occlusion, camera movement, and interactions without necessarily recording the controls that produced them. Self-supervised learning can exploit these sequences by predicting missing visual information or subsequent observations. Passive prediction, however, differs from action-conditioned simulation: predicting what usually follows a scene does not specify what would happen if an agent selected a particular action. Video-trained systems address this distinction either by inferring a controllable latent action space from observations or by adding interaction data after visual pretraining. Genie and V-JEPA 2 exemplify these two approaches. (arxiv.org)\n\nThe original Genie, introduced by Jake Bruce and colleagues at Google DeepMind in February 2024, was an approximately 11-billion-parameter research model trained on a filtered collection of 30,000 hours of videos from hundreds of two-dimensional platform games, without recorded action labels. Its three components were a spatiotemporal video tokenizer, a latent-action model, and an autoregressive dynamics model. The tokenizer compressed frames into discrete visual tokens using spatial and temporal attention. During training, the latent-action model examined previous frames and the subsequent frame, encoding their change through a bottleneck with eight discrete action codes. The dynamics model learned to predict the next frame’s tokens from preceding tokens and inferred actions; generation was autoregressive across frames, with iterative masked-token sampling within each frame. (arxiv.org)\n\nAt inference time, Genie’s action-inference network was replaced by user-selected action codes. A prompt image was tokenized, an action was supplied, and the predicted tokens were decoded into the next image; repeating this process provided frame-by-frame control. Demonstrations used synthetic images, photographs, and hand-drawn sketches as starting scenes, with the same latent codes producing similar movements across different prompts. Text could enter indirectly through a text-to-image model that supplied the initial frame. Such conditioning creates an interactive continuation consistent with learned visual regularities, rather than establishing a faithful reconstruction of the photographed location or the rules of a particular existing video game. (arxiv.org)\n\nV-JEPA 2, presented by Mido Assran and colleagues in June 2025, instead used predictive representation learning. Pretraining drew on more than one million hours of internet video, alongside image data. An encoder processed visible regions, while a predictor estimated representations of masked spatiotemporal regions produced by a target encoder. The prediction target was therefore a feature representation, not reconstructed pixels. Its action-conditioned extension, V-JEPA 2-AC, retained the pretrained encoder and trained a predictor using less than 62 hours of DROID robot data. Although no task annotations were required, this adaptation was not based on video alone: accompanying end-effector positions, orientations, and gripper states supplied seven-dimensional state vectors, and differences between consecutive states defined actions. (arxiv.org)\n\nVisual prompts, actions, and goals play different roles in these systems. A Genie prompt initializes a generated scene, whereas successive action codes steer its continuation. In V-JEPA 2-AC, candidate robot-action sequences condition predicted future features, and a goal image supplies a target representation against which those predictions can be compared. The latter supports goal-directed planning without generating a viewable future video. Neither a visually convincing continuation nor proximity in feature space alone establishes that every underlying physical variable has been recovered. (sites.google.com)\n\nConverting observation into controllable dynamics leaves several specific difficulties. Action identifiability concerns whether an inferred code represents an agent’s intervention rather than another source of visible change; Genie’s latent controls are learned codes, not automatically identified keyboard commands or robot actuations. Hidden state presents a separate problem: an image can omit velocities, occluded objects, or geometry that affects subsequent motion, motivating the use of temporal context. Long-term consistency additionally requires preserving earlier objects and changes while repeatedly feeding generated predictions back into the model. These are distinct requirements from producing plausible individual frames; autoregressive interactive generation can accumulate inconsistencies as interaction continues. (arxiv.org)" }, { "heading": "Applications and demonstrated capabilities", "text": "In discrete strategic games, learned models have supported planning without a supplied transition simulator. Julian Schrittwieser and colleagues’ MuZero, published in Nature in 2020, matched AlphaZero’s superhuman performance in Go, chess and shogi and achieved state-of-the-art results across 57 Atari games. Its model predicted rewards, policies and values for search rather than reconstructing future images. These results concern an agent combining a learned model with tree search, not a predictive model acting alone. (nature.com) Continuous-control demonstrations address different demands: actions vary continuously, and success depends on coordinating movement under evolving physical dynamics. PlaNet’s 2019 evaluation used six image-based simulated tasks, including cheetah running, walker walking, finger spinning and catching a ball in a cup, with actions selected by planning in latent space. These were visual locomotion and manipulation benchmarks, not physical-robot deployments. (proceedings.mlr.press)\n\nDanijar Hafner, Jurgis Pasukonis, Jimmy Ba and Timothy Lillicrap’s third-generation Dreamer evaluation, published on 2 April 2025, covered more than 150 tasks using one hyperparameter configuration. Its benchmarks included 20 continuous-control tasks evaluated separately with proprioceptive inputs and with image-only observations, spanning classical control, locomotion and simulated robot manipulation. The reported breadth concerned the same learning algorithm applied across tasks, rather than one trained policy performing every task without further learning. (nature.com) In Minecraft, Dreamer learned the resource-gathering and crafting sequence leading to diamond collection from pixels and sparse rewards, without human demonstrations or curricula. All evaluated Dreamer training runs discovered diamonds within 100 million environment steps. The experiment nevertheless supplied a task reward structure and used a categorical action interface containing abstract crafting actions; block breaking was accelerated. Thus “from scratch” described the absence of demonstration-based learning and staged curricula, not an absence of engineered task conditions. (nature.com)\n\nPhysical robotic manipulation was demonstrated by V-JEPA 2-AC in 2025. Following video pretraining, the action-conditioned model was post-trained with less than 62 hours of robot interaction video from DROID and deployed on Franka arms in two laboratories without collecting training data from those deployment environments or using task-specific rewards. (arxiv.org) A planner selected actions by minimizing the distance between predicted future representations and an image-goal representation, executing an action and then replanning. Pick-and-place used three visual subgoals: grasping the object, moving it near the destination and placing it. Across the two laboratories, average pick-and-place success was 80% for cups and 65% for boxes, with ten trials per laboratory and object condition. These were bounded tabletop demonstrations: camera positions were manually selected, and image subgoals helped manage the difficulty of long-horizon planning. (arxiv.org)\n\nGenerated interactive environments provide a separate setting for behavioural experimentation. In December 2024, Google DeepMind demonstrated its SIMA agent following instructions inside Genie 2-generated environments, including opening either a blue or a red door. This tested an existing agent’s instruction-following behaviour in newly generated scenes; it was not, by itself, evidence that training in those scenes improved performance elsewhere. The accompanying proposal to generate diverse training and evaluation worlds extended beyond that demonstration. (deepmind.google) The original 2024 Genie paper demonstrated a related but distinct training use: its learned latent-action model labelled previously unseen expert videos for imitation learning in CoinRun. With as few as 200 action-labelled expert samples to map latent controls to real controls, the resulting policy matched an action-supervised behavioural-cloning baseline. That experiment trained from inferred actions in videos, rather than establishing general-purpose reinforcement learning wholly inside generated worlds. (arxiv.org)\n\nThese demonstrations consequently establish different kinds of capability. MuZero and Dreamer measured decision-making in software-defined games and simulators; Genie-based experiments examined controllability, imitation or agent behaviour in learned environments; V-JEPA 2-AC measured closed-loop manipulation on physical hardware. Neither visually convincing interaction nor high simulated reward establishes physical reliability, and successful tabletop manipulation does not establish unrestricted competence across objects, environments or longer tasks. (nature.com)" }, { "heading": "Evaluation and experimental methodology", "text": "Evaluation separates five questions: whether predictions match observed outcomes; whether model-assisted decisions improve return or task success; how much environmental interaction learning requires; how much computation training and deployment consume; and whether performance transfers to unfamiliar conditions. These quantities are not interchangeable. PlaNet’s 2019 experiments, for example, measured both control performance and episodes required to learn, rather than treating accurate image reconstruction as sufficient evidence of useful dynamics. A model intended for planning must preserve action-dependent rewards and outcomes, even when some visual details are irrelevant to the task. (proceedings.mlr.press)\n\nPredictive tests differ according to the information supplied during evaluation. Held-out one-step prediction conditions on recorded observations and actions and predicts the next outcome in a test set. Open-loop multi-step prediction starts from an observed history and recursively advances the model without incorporating subsequent ground-truth observations. Closed-loop evaluation lets a policy choose actions from evolving observations; this can test either an agent in the actual environment or a policy interacting with a learned simulator, and the distinction must remain explicit. One-step accuracy can conceal compounding errors, while replanning from fresh observations can compensate for inaccuracies that become severe in uninterrupted rollouts. Conversely, policies can visit states absent from the recorded test trajectories. These tests therefore need not rank models identically. (arxiv.org)\n\nAction responsiveness requires more than visually plausible sequences. A controlled test holds the starting state or history fixed, varies the action sequence, and compares predicted changes with executed outcomes—for example, object displacement, contact, or end-effector motion. Reward-equipped models can additionally be assessed through reward error, predicted cumulative return, and agreement with the actual ranking of candidate policies. Task-success measurements then establish whether model-based choices work outside the model. A 2026 methodological position paper proposed combining these diagnostics with tests of the gap between predicted and realized performance for policies optimized against the model; it presented this as an evaluation framework, not an established universal benchmark. (arxiv.org)\n\nBenchmark protocols specify what counts as interaction and what information is available. Atari100k permits 100,000 agent decisions, conventionally corresponding to 400,000 game frames with action repeat four; reporting only “steps” obscures that distinction. Comparisons also depend on pixel versus state observations, discrete versus continuous controls, episode termination, demonstration availability, and external pretraining. Hafner and colleagues’ 2025 Dreamer study used one hyperparameter configuration across eight domains and more than 150 tasks. Its default protocol used five independent training seeds, with ten for BSuite and Minecraft, and plotted means with one standard deviation. This tests robustness of a fixed configuration rather than performance after separate task-specific tuning. (nature.com) For statistical interpretation, Agarwal and colleagues’ 2021 study of reinforcement learning benchmarks showed that rankings based on point estimates could change under uncertainty analysis, motivating interval estimates and robust aggregates such as the interquartile mean. (papers.nips.cc)\n\nInteraction efficiency is measured by learning curves against real transitions or frames, not imagined transitions. Computational expense requires separate reporting of hardware, elapsed training time, update frequency, and inference or planning latency. Dreamer’s 2025 experiments trained each agent on one Nvidia A100 GPU, but that hardware statement alone does not establish equal computational cost between algorithms. (nature.com) The 2025 V-JEPA 2 study illustrates the importance of deployment latency: its Cosmos comparison required approximately four minutes per planned action on an RTX 4090 under the reported sampling configuration, making a complete pick-and-place execution exceed an hour. Success rate without timing would omit a consequential experimental difference. (arxiv.org)\n\nAblations isolate contributions by replacing representations, removing objectives, or changing planning components while retaining comparable data and evaluation budgets. PlaNet compared deterministic, stochastic, and combined dynamics and examined latent overshooting, separating architectural choices from multi-step training effects. (proceedings.mlr.press) Generalization tests separately withhold levels, objects, tasks, or deployment settings; training a fresh agent on every benchmark task is different from transferring a pretrained model. V-JEPA 2-AC’s robotic evaluation used unfamiliar objects and environments without collecting training data in those deployment environments. Its ablations also examined video pretraining, action conditioning, and the training rollout horizon, connecting representation and prediction choices to measured robot success rather than visual quality alone. (arxiv.org)" }, { "heading": "Connections to cognition and open research questions", "text": "Imagined rollouts provide a functional analogy to internal simulation in cognitive science: an agent represents a situation, projects possible consequences, and uses those projections without first executing every action externally. In David Ha and Jürgen Schmidhuber’s 2018 World Models, a controller could be trained inside a learned recurrent environment and subsequently transferred to the original game environment. Here, “dreaming” denotes sampling sequences from a neural dynamics model, not biological dreaming. The engineering result establishes that synthetic experience can support policy learning; it does not establish that humans or animals use the same representations, learning objectives, or rollout procedures. Connections to cognition therefore concern computational functions rather than demonstrated equivalence between neural-network components and biological mechanisms. (arxiv.org)\n\nHierarchical prediction attempts to replace lengthy sequences of elementary transitions with predictions at several spatial or temporal scales. A higher-level representation might describe reaching a location, while a lower-level model describes movements needed to reach it. Yann LeCun’s 2022 position paper A Path Towards Autonomous Machine Intelligence proposed hierarchical joint-embedding predictive architectures in which higher levels set objectives for lower levels. A distinct implemented example is Director, introduced by Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel in 2022: its manager selects latent goals every eight time steps, while a worker chooses elementary actions. Director learns these policies through imagined experience in a single fine-grained world model; hierarchical control is therefore not identical to learning multiple levels of dynamics. Such architectures are research approaches to long-horizon behaviour, not established requirements for intelligence. (openreview.net)\n\nAccurate observational prediction must also be distinguished from causal inference. In general, \\(p(o_{t+1}\\mid o_t,a_t)\\), estimated from recorded behaviour, need not equal \\(p(o_{t+1}\\mid o_t,\\operatorname{do}(a_t))\\), which describes externally setting an action. If an unobserved variable influences both the recorded action and the subsequent outcome, an action can predict that outcome without causing it. Conditioning on an adequately informative state and obtaining suitable intervention data can address this distinction, but merely adding an action input does not establish those conditions. Pim de Haan, Dinesh Jayaraman, and Sergey Levine’s 2019 Causal Confusion in Imitation Learning demonstrated the related problem that policies can exploit correlates of an expert’s actions and fail when their own behaviour changes those correlations. For world models, this motivates testing consequences under interventions rather than equating passive forecasting accuracy with reliable action prediction. (arxiv.org)\n\nIntegration with language models provides another interface to predictive representations, rather than making language prediction itself a physical simulator. The 2025 V-JEPA 2 paper connected visual embeddings to a pretrained language model through a learned projector and trained the combined system on image–text and video–text pairs. Its scaled configuration used Llama 3.1 8B and an 88.5-million-sample alignment dataset, reporting 84.0 on PerceptionTest and 76.9 on TempCompass. These video question-answering results concern interpreting visual information and producing linguistic answers. The paper’s action-conditioned robot predictor, V-JEPA 2-AC, was a separate adaptation. This separation illustrates that multimodal learning, linguistic reporting, and controllable dynamics prediction can share representations while retaining different objectives and evidential requirements. (arxiv.org)\n\nReusable models raise questions about what should remain invariant across tasks and what must change across environments. V-JEPA 2’s observational pretraining followed by action-conditioned adaptation offers one concrete approach, but leaves open how broadly a representation can support different actuators, observation systems, and interaction dynamics. LeCun’s configurable-model proposal addresses a related issue: retaining every sensory detail is different from retaining the variables needed for a particular decision. Open problems include learning abstractions that remain predictive over long horizons, adapting continually without losing previously acquired prediction and control skills, and identifying when discarded detail becomes important for a new task. Ha and Schmidhuber’s 2018 discussion explicitly identified behavioural replay as a possible mechanism for preserving old skills, while leaving experiments with those broader approaches to future work. (arxiv.org)" } ], "links": [ "artificial-intelligence", "latent-space", "reinforcement-learning", "control-theory", "generative-artificial-intelligence", "representation-learning", "policy-reinforcement-learning", "value-function", "knowledge-base", "variational-autoencoder", "recurrent-neural-network", "markov-property", "markov-decision-process", "bayesian-inference", "encoder-decoder-architecture", "self-supervised-learning", "training-data", "loss-function", "variational-inference", "evidence-lower-bound", "kullback-leibler-divergence", "ensemble-learning", "distribution-shift", "mathematical-optimization", "generalization", "attention-mechanism", "video-game", "game-tree", "robotics", "test-set", "cognitive-science", "causal-inference", "language-model", "multimodal-learning" ], "sources": [ { "title": "World Models", "url": "https://worldmodels.github.io/" }, { "title": "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model", "url": "https://arxiv.org/abs/1911.08265" }, { "title": "Genie: Generative Interactive Environments", "url": "https://arxiv.org/abs/2402.15391" }, { "title": "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning", "url": "https://arxiv.org/abs/2506.09985" }, { "title": "Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming", "url": "https://mlanthology.org/icml/1990/sutton1990icml-integrated/" }, { "title": "World Models", "url": "https://arxiv.org/abs/1803.10122" }, { "title": "Learning Latent Dynamics for Planning from Pixels", "url": "https://arxiv.org/abs/1811.04551" }, { "title": "Learning Latent Dynamics for Planning from Pixels", "url": "https://proceedings.mlr.press/v97/hafner19a" }, { "title": "Dream to Control: Learning Behaviors by Latent Imagination", "url": "https://arxiv.org/abs/1912.01603" }, { "title": "Dream to Control: Learning Behaviors by Latent Imagination", "url": "https://openreview.net/pdf?id=S1lOTC4tDS" }, { "title": "Mastering Diverse Domains through World Models", "url": "https://arxiv.org/abs/2301.04104" }, { "title": "Mastering diverse control tasks through world models", "url": "https://www.nature.com/articles/s41586-025-08744-2" }, { "title": "Planning and Acting in Partially Observable Stochastic Domains", "url": "https://www.cassandra.org/arc/papers/aij98.pdf" }, { "title": "Learning Latent Dynamics for Planning from Pixels", "url": "https://proceedings.mlr.press/v97/hafner19a/hafner19a.pdf" }, { "title": "Mastering Diverse Domains through World Models", "url": "https://arxiv.org/html/2301.04104v2" }, { "title": "Genie: Generative Interactive Environments", "url": "https://arxiv.org/html/2402.15391v1" }, { "title": "V-JEPA: Video Joint Embedding Predictive Architecture", "url": "https://github.com/facebookresearch/jepa/blob/main/README.md" }, { "title": "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning", "url": "https://arxiv.org/html/2506.09985v1" }, { "title": "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model", "url": "https://arxiv.org/pdf/1911.08265" }, { "title": "Mastering Atari, Go, chess and shogi by planning with a learned model", "url": "https://www.nature.com/articles/s41586-020-03051-4" }, { "title": "Dream to Control: Learning Behaviors by Latent Imagination", "url": "https://arxiv.org/html/1912.01603v3" }, { "title": "When to Trust Your Model: Model-Based Policy Optimization", "url": "https://papers.neurips.cc/paper/9416-when-to-trust-your-model-model-based-policy-optimization.pdf" }, { "title": "Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models", "url": "https://arxiv.org/abs/1805.12114" }, { "title": "When to Trust Your Model: Model-Based Policy Optimization", "url": "https://papers.nips.cc/paper_files/paper/2019/file/5faf461eff3099671ad63c6f3f094f7f-Paper.pdf" }, { "title": "MOPO: Model-based Offline Policy Optimization", "url": "https://proceedings.neurips.cc/paper/2020/file/a322852ce0df73e204b7e67cbbef0d0a-Paper.pdf" }, { "title": "🧞 Genie: Generative Interactive Environments", "url": "https://sites.google.com/view/genie-2024/home" }, { "title": "Introducing V-JEPA 2", "url": "https://ai.meta.com/research/vjepa" }, { "title": "Genie 3 — Google DeepMind", "url": "https://deepmind.google/models/genie/" }, { "title": "Learning Latent Dynamics for Planning from Pixels: Supplementary Material", "url": "https://proceedings.mlr.press/v97/hafner19a/hafner19a-supp.pdf" }, { "title": "Genie 2: A large-scale foundation world model", "url": "https://deepmind.google/blog/genie-2-a-large-scale-foundation-world-model/?trk=article-ssr-frontend-pulse_little-text-block" }, { "title": "How Should World Models Be Evaluated? A Decision-Making-Centric Position", "url": "https://arxiv.org/abs/2606.15032" }, { "title": "Deep Reinforcement Learning at the Edge of the Statistical Precipice", "url": "https://papers.nips.cc/paper/2021/file/f514cec81cb148559cf475e7426eed5e-Paper.pdf" }, { "title": "World Models", "url": "https://worldmodels.github.io/?trk=article-ssr-frontend-pulse_little-text-block" }, { "title": "A Path Towards Autonomous Machine Intelligence", "url": "https://openreview.net/pdf/315d43ba26f55357a84cec9a7ed15a6610094f79.pdf" }, { "title": "Deep Hierarchical Planning from Pixels", "url": "https://papers.nips.cc/paper_files/paper/2022/file/a766f56d2da42cae20b5652970ec04ef-Paper-Conference.pdf" }, { "title": "Causal Confusion in Imitation Learning", "url": "https://arxiv.org/abs/1905.11979" } ], "license": "https://creativecommons.org/licenses/by/4.0/" }