Two-axis trigger
Repeated sampling and verification add computation, so the trigger allocates this cost to states where it is most useful. It is designed to capture two complementary conditions: contact-rich interactions, where manipulation is more complex and failures tend to occur, and uncertain states, where different samples provide meaningful alternatives for the verifier.
Two initial action chunks provide both signals. The temporal axis measures changes between consecutive actions within each chunk, capturing motion variation associated with challenging interactions. The sample axis measures disagreement between the two chunks, capturing policy uncertainty. Each chunk is normalized by its average action magnitude to avoid favoring fast-moving stages.
Latent world model verifier
Action selection requires estimating the consequences of each candidate. A JEPA-based world model predicts action-conditioned future visual latents and task progress in a shared latent space, avoiding the cost of reconstructing future images. Once triggered, the policy expands the pool to 10 candidates, and the verifier selects the action chunk with the largest cumulative predicted progress.
Each action chunk is divided into sub-chunks for multi-step latent prediction. This provides denser supervision than predicting only the final state, while reducing inference cost and error accumulation compared with predicting after every individual action.
Progress estimation also requires task information: the same screen-lifting action can advance “opening the computer” while reversing “closing the computer.” A task-specific learnable query conditions the progress prediction branch, allowing the verifier to evaluate future states relative to the instructed task.
The attention mask prevents latent state prediction from attending to the task query. Without this restriction, task information could leak into the dynamics model and encourage shortcuts based on task semantics rather than the physical consequences of the action. The progress branch can use task information to judge whether a transition is useful, while the latent prediction branch models the environment dynamics independently of the task. This separation adapts the pretrained world model for verification while preserving its original modeling objective.
Task query and attention mask
Opening the computer gives positive progress for the screen-lifting action. The mask keeps task information out of latent dynamics prediction.