Pixel-level Scene Understanding in One Token: Visual States Need What-is-Where Composition

Researchers propose CroBo, a new visual state representation learning framework that helps robotic agents better understand dynamic environments by encoding both semantic identities and spatial locations of scene elements. The framework uses a global-to-local reconstruction method that compresses observations into compact tokens, achieving state-of-the-art performance on robot policy learning benchmarks.

• →CroBo framework addresses the challenge of learning visual states from streaming video for robotic decision making.
• →The method captures 'what-is-where' by jointly encoding semantic identities and spatial locations of scene elements.
• →Uses global-to-local reconstruction with heavily masked patches and sparse visible cues for learning.
• →Achieves state-of-the-art performance on diverse vision-based robot policy learning benchmarks.
• →Learned representations preserve pixel-level scene composition and track element movement over time.