UNCOM: Zero-shot Context-Aware Command Understanding for Tabletop Scenarios

UNCOM is a zero-shot framework that enables robots to understand natural human commands in tabletop environments by integrating speech, gestures, and scene context without requiring task-specific training data. The system achieves 82.39% success rate on real-world interaction scenarios, demonstrating practical viability for general-purpose domestic robotics applications.

UNCOM addresses a fundamental challenge in human-robot interaction: enabling machines to understand contextually-grounded commands without extensive task-specific training. The framework's zero-shot approach is significant because it reduces deployment friction for robotics applications, allowing systems to operate across diverse environments and scenarios without retraining. By leveraging foundational models for speech recognition, natural language understanding, gesture detection, and object segmentation, UNCOM achieves capability that would have required substantial labeled datasets just years ago.

The technical approach reflects broader trends in AI where foundation models trained on diverse internet-scale data provide general-purpose capabilities that downstream applications can leverage. The explicit object-action-target parsing enhances interpretability, addressing growing concerns about AI system transparency in safety-critical robotics applications. This modular design enables integration with symbolic reasoning frameworks, bridging neural and symbolic AI approaches.

From an industry perspective, this work validates that practical human-robot collaboration in domestic settings becomes feasible as foundation models mature. The 82.39% success rate on real-world data suggests the system handles genuine complexity, noise, and ambiguity rather than simplified benchmarks. This has implications for robotics manufacturers evaluating deployment strategies and for smart home ecosystems considering robot integration.

The public release of datasets, code, and evaluation scenarios accelerates research velocity in human-robot interaction. Future work will likely focus on improving multimodal integration, handling more complex spatial reasoning, and extending beyond tabletop scenarios. The framework's success indicates that combining foundational models with task-specific architectural choices produces robust systems for practical robotics applications.

• →UNCOM achieves 82.39% success on real-world human-robot interaction without task-specific training data.
• →Zero-shot approach leverages foundation models to reduce deployment friction for robotics applications.
• →Explicit object-action-target parsing improves interpretability and enables symbolic reasoning integration.
• →Multimodal fusion of speech, gestures, and scene context handles real-world communication ambiguity.
• →Public release of code and datasets accelerates research progress in practical human-robot collaboration.