LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

Researchers introduced LakeQA, a new benchmark dataset for evaluating large language models on question-answering tasks over massive data lakes containing 9.5TB of heterogeneous data. The benchmark reveals significant challenges in current LLMs, with GPT-5.2 achieving only 18.37% accuracy, highlighting the gap between reading-comprehension performance and real-world search-and-reasoning requirements.

LakeQA addresses a critical limitation in current LLM evaluation frameworks. While existing benchmarks typically provide explicit evidence documents or require only trivial retrieval, real-world applications demand agents capable of searching through massive unstructured data repositories and synthesizing multi-hop reasoning across disparate sources. This gap between benchmark realism and practical requirements has obscured the true capabilities and limitations of frontier language models.

The benchmark's construction reflects rigorous academic standards, incorporating 9.5TB of Wikipedia and government data annotated by Ph.D.-level experts. Each task requires agents to discover relevant documents autonomously before composing coherent answers—a workflow mirroring production search systems. The heterogeneous data structure mirrors actual enterprise data lakes where information exists across structured databases, unstructured text, and semi-structured formats.

The performance results carry significant implications for AI development trajectories. GPT-5.2's 18.37% exact-match rate demonstrates that scaling model parameters alone proves insufficient for search-centric reasoning. This suggests future progress demands architectural innovations in retrieval mechanisms, reasoning chains, and information synthesis rather than incremental improvements to existing approaches. The benchmark effectively decouples reading comprehension (where LLMs excel) from discovery and integration (where they struggle).

Developers building production AI systems now have a realistic testing ground that validates search capabilities alongside reasoning. This standardized benchmark enables comparative analysis across model architectures and prompting strategies, potentially driving research toward hybrid systems combining neural retrieval with reasoning components. Organizations evaluating LLMs for data analysis applications should expect significantly lower performance than reported on simpler benchmarks.

• →LakeQA benchmark reveals frontier LLMs achieve only ~18% accuracy on realistic search-and-reasoning tasks despite strong reading-comprehension performance
• →The benchmark uses 9.5TB of heterogeneous data from Wikipedia and government sources, requiring multi-hop reasoning across implicit intermediate steps
• →Current LLM evaluation frameworks inadequately test real-world requirements for searching massive data lakes and synthesizing evidence across sources
• →Results suggest scaling model parameters alone is insufficient for data lake QA, pointing toward needs for improved retrieval and reasoning architectures
• →The benchmark provides standardized testing for developing LLM agents in production data analysis applications across enterprise environments