Anonymous Submission
Video generative models are an emerging tool for embodied agents: they can serve as scalable visual planners, generating videos that depict how to complete previously unseen tasks. However, executing these plans typically requires a video-to-action or "inverse dynamics" model trained on large, human-collected video-action datasets, severely limiting scalability. We introduce Mirror, a self-supervised approach that learns video-to-action mappings through environment interactions, without prior action data or hand-crafted rewards. Mirror learns by attempting to follow a visual plan by executing its best guess of the underlying actions in the environment. Mirror then trains on the resulting video-action pairs, incrementally self-improving and following the visual planner more closely. Across Atari, OGBench, and DeepMind Control, Mirror outperforms leading learning-from-observation methods with learned video plans, and achieves near-expert performance with oracle plans. On Minecraft BASALT, it outperforms methods with access to expert actions or hand-engineered task knowledge. In Minecraft survival, it is the first method to acquire an iron pickaxe through self-improvement alone, without rewards or expert action labels. In the offline setting, with expert action labels but no environment interaction, it sets a new state of the art in obtaining diamonds. Our results suggest that advances in video models can translate into increasingly capable agents without a corresponding need for hand-crafted rewards or large collections of expert action data.
In the revision, we retrained our generative video planner exclusively on human gameplay video from the VPT dataset. Without action labels or rewards, Mirror now crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3. In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4.
Each clip shows the video plan (left) beside Mirror's execution (right). The planner replans from the agent's actual observation, so the clips also show the agent recovering from its mistakes. The bar under each clip marks where the agent reaches each milestone; click a milestone to jump to it.
We evaluate Grounding on the Minecraft Diamond Challenge. Following the protocol of the original paper, we generate 7 subgoal frames and use them to train a goal-conditioned policy. Grounding's policy struggles to follow the subgoals. The agent moves in seemingly random directions.
Replanning after every subgoal is critical. A baseline where we generate single-image subgoals and replan after every 16 steps is able to navigate towards a tree and break it. However, the agent's actions are still noisy due to the high dimensionality of the action space.
Both Grounding and the single-image subgoal baseline underperform compared to our method. Dense video planning and replanning allows the agent to explore the action space efficiently, enabling it to solve long-horizon tasks.
Mirror on the Minecraft BASALT Waterfall task.
A comparison of Mirror on the Minecraft BASALT FindCave task over time.
Grids of results on the Minecraft BASALT FindCave and Waterfall tasks.