HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities

Researchers introduce HOME-KGQA, a new benchmark dataset for evaluating knowledge graph question answering systems on household activities using multimodal data. The dataset reveals significant performance gaps in current LLM-based KGQA methods, highlighting critical challenges for real-world deployment of AI systems that combine language models with structured knowledge.

HOME-KGQA addresses a fundamental limitation in how AI systems are evaluated and deployed in practical settings. While existing KGQA benchmarks focus on encyclopedic knowledge from sources like Wikipedia, real-world applications require understanding spatiotemporal relationships within specific domains like household environments. This gap between benchmark performance and real-world capability represents a critical blind spot in AI development, where models appear competent in controlled settings but struggle with concrete, grounded reasoning tasks.

The research emerges from growing recognition that Large Language Models hallucinate when operating without access to structured external knowledge. Knowledge graphs provide explicit, verifiable facts, making them valuable for reducing false outputs. However, integrating these approaches effectively requires benchmarks that reflect actual deployment scenarios—not just question-answering over static databases. The multimodal aspect (combining visual, temporal, and structural data) adds another layer of complexity absent from existing datasets.

For developers and researchers building embodied AI systems, this work signals that current methods lack the sophistication needed for real-world reliability. The performance gap between existing benchmarks and HOME-KGQA demonstrates that production-grade KGQA systems require substantial architectural improvements beyond scaling existing approaches. Organizations investing in AI systems for robotics, smart home applications, or other embodied contexts need to account for these limitations in their technical roadmaps and evaluation strategies.

The public release of HOME-KGQA creates opportunities for the research community to develop more robust solutions. Future work will likely focus on hybrid architectures that better handle complex spatiotemporal reasoning while maintaining the verifiability that knowledge graphs provide.

• →Current LLM-based KGQA methods significantly underperform on real-world household activity datasets compared to encyclopedic knowledge benchmarks.
• →Multimodal knowledge graphs combining visual, temporal, and structural data require fundamentally different evaluation approaches than existing single-modality datasets.
• →The performance gap highlights that production-ready AI systems for embodied applications need substantial architectural improvements beyond current methods.
• →Knowledge graphs remain essential for reducing hallucinations, but integrating them effectively requires benchmarks reflecting concrete, grounded reasoning tasks.
• →HOME-KGQA's public release enables systematic research into hybrid architectures for handling complex spatiotemporal reasoning in structured knowledge systems.