Learning Physical Principles from Interaction: Self-Evolving Planning via Test-Time Memory

Researchers introduce PhysMem, a memory framework that enables vision-language model robot planners to learn physical principles through real-time interaction without updating model parameters. The system records experiences, generates hypotheses, and verifies them before application, achieving 76% success on brick insertion tasks compared to 23% for direct experience retrieval.

• →PhysMem allows VLM robot planners to learn physical properties through test-time interaction without parameter updates.
• →The system uses a verification-before-application approach to test hypotheses against new observations.
• →Performance improved significantly with 76% success rate versus 23% for direct experience retrieval on controlled tasks.
• →The framework was evaluated across four VLM backbones on real-world manipulation tasks and simulations.
• →Real-world experiments showed consistent improvement over 30-minute deployment sessions.