Self-Improving Video-to-Action Models

Anonymous Submission

TL;DR: Mirror solves embodied tasks without expert action labels or task rewards through repeated self-play with a large-scale video planner.

Abstract

Video generative models are an emerging tool for embodied agents: they can serve as scalable visual planners, generating videos that depict how to complete previously unseen tasks. However, executing these plans typically requires a video-to-action or "inverse dynamics" model trained on large, human-collected video-action datasets, severely limiting scalability. We introduce Mirror, a self-supervised approach that learns video-to-action mappings through environment interactions, without prior action data or hand-crafted rewards. Mirror learns by attempting to follow a visual plan by executing its best guess of the underlying actions in the environment. Mirror then trains on the resulting video-action pairs, incrementally self-improving and following the visual planner more closely. Across Atari, OGBench, and DeepMind Control, Mirror outperforms leading learning-from-observation methods with learned video plans, and achieves near-expert performance with oracle plans. On Minecraft BASALT, it outperforms methods with access to expert actions or hand-engineered task knowledge. In Minecraft survival, it is the first method to acquire an iron pickaxe through self-improvement alone, without rewards or expert action labels. In the offline setting, with expert action labels but no environment interaction, it sets a new state of the art in obtaining diamonds. Our results suggest that advances in video models can translate into increasingly capable agents without a corresponding need for hand-crafted rewards or large collections of expert action data.

New Rebuttal Results: Minecraft Diamond Challenge

In the revision, we retrained our generative video planner exclusively on human gameplay video from the VPT dataset. Without action labels or rewards, Mirror now crafts an iron pickaxe in 15.6% of episodes, against 1.7% for DreamerV3. In Dreamer 4's offline setting, it obtains diamonds in 5.5% of episodes, against 0.7% for Dreamer 4.

Each clip shows the video plan (left) beside Mirror's execution (right). The planner replans from the agent's actual observation, so the clips also show the agent recovering from its mistakes. The bar under each clip marks where the agent reaches each milestone; click a milestone to jump to it.

Video plan Execution1×

Success rates across the twelve milestones of the Diamond Challenge. Offline methods learn from VPT human data without environment interaction. Online methods interact with the environment for about 10M steps (~140 in-game hours) without expert action labels. Mirror is evaluated on 128 episodes of 60 minutes with respawning, following VPT and Dreamer 4. Offline and online baselines are from Dreamer 4 and DreamerV3, respectively.

New Rebuttal Results: Grounding on Minecraft with an Oracle VP

Grounding - 1M Environment Steps

We evaluate Grounding on the Minecraft Diamond Challenge. Following the protocol of the original paper, we generate 7 subgoal frames and use them to train a goal-conditioned policy. Grounding's policy struggles to follow the subgoals. The agent moves in seemingly random directions.

Single Image Subgoal with Replanning - 1M Environment Steps

Replanning after every subgoal is critical. A baseline where we generate single-image subgoals and replan after every 16 steps is able to navigate towards a tree and break it. However, the agent's actions are still noisy due to the high dimensionality of the action space.

Mirror (Ours) - 1M Environment Steps

Both Grounding and the single-image subgoal baseline underperform compared to our method. Dense video planning and replanning allows the agent to explore the action space efficiently, enabling it to solve long-horizon tasks.

Waterfall Results

Mirror on the Minecraft BASALT Waterfall task.

Episode 0

Episode 1: Failure Case

Episode 2

Episode 3

FindCave Analysis Over Time

A comparison of Mirror on the Minecraft BASALT FindCave task over time.

OGBench

AntMaze Large

Cube Double

DeepMind Control

Walker Walk

Atari

Pong

In-Context Learning: Lava Escape

Results Grids

Grids of results on the Minecraft BASALT FindCave and Waterfall tasks.

FindCave

Waterfall