See, Act, Adapt: Active Perception for Unsupervised Cross-Domain Visual Adaptation via Personalized VLM-Guided Agent

Researchers introduce Sea² (See, Act, Adapt), a novel approach that improves AI perception models in new environments by using an intelligent pose-control agent rather than retraining the models themselves. The method keeps perception modules frozen and uses a vision-language model as a controller, achieving significant performance improvements of 13-27% across visual tasks without requiring additional training data.

• →Sea² addresses the problem of AI perception models degrading in novel environments like indoor scenes without requiring costly retraining or annotations.
• →The approach uses a frozen perception module with an intelligent agent that navigates to informative viewpoints based on scalar feedback.
• →A vision-language model is transformed into a pose controller through rule-based exploration followed by unsupervised reinforcement learning.
• →Performance improvements range from 13.54% to 27.68% across visual grounding, segmentation, and 3D box estimation tasks.
• →The method eliminates catastrophic forgetting and works with off-the-shelf perception models without model-specific coupling.