PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?

Researchers introduce PrepBench, a new benchmark for evaluating how well large language models can handle natural language-driven data preparation tasks. The benchmark reveals that despite recent LLM advances, current models still struggle significantly with translating user intent into executable data preparation workflows, particularly when handling ambiguous requirements and complex real-world datasets.

PrepBench addresses a critical gap in AI evaluation by focusing on practical data preparation workflows rather than generic code generation. The benchmark captures three essential capabilities—interactive disambiguation, code generation, and workflow translation—that existing benchmarks overlook. This matters because data preparation consumes substantial time in enterprise data analysis pipelines, and automating it through natural language interfaces could unlock significant productivity gains.

The benchmark construction reveals the depth of this challenge. Tasks drawn from Preppin' Data Challenges contain 3-18 preparation steps, with nearly half requiring over 100 lines of Python code. This complexity mirrors real-world scenarios where data is messy, user intent is ambiguous, and solutions demand iterative refinement. Commercial tools have traditionally relied on GUI-based workflows precisely because this work is complex and context-dependent.

The research demonstrates that state-of-the-art LLMs currently fall short of handling this paradigm shift effectively. This finding carries implications for enterprises expecting to replace data engineering workflows with natural language interfaces. The gap suggests that LLM-based data preparation assistants will likely function as augmentation tools rather than autonomous replacements in the near term, requiring human validation and intervention.

Looking forward, PrepBench establishes a standardized measurement framework for tracking progress in this domain. The benchmark's design—incorporating real-world complexity and iterative validation requirements—should drive more targeted model development and highlight specific capability gaps. Organizations considering LLM investments for data preparation should recognize this work as an honest assessment of current limitations rather than a roadmap to immediate automation.

• →PrepBench benchmark reveals state-of-the-art LLMs struggle with natural language-driven data preparation despite recent advances
• →Current models face challenges with ambiguous intent disambiguation and translating code into interpretable, validatable workflows
• →Nearly half of benchmark tasks require over 100 lines of Python code, reflecting real-world complexity that exceeds simple code-generation scenarios
• →The research suggests LLM-based data prep tools will function as augmentation aids rather than autonomous replacements in near term
• →The benchmark provides a standardized framework for measuring progress toward NL-driven data preparation systems