Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

Researchers demonstrate using large language models to automate the generation of increasingly difficult benchmark instances for testing neural reasoning systems. The approach combines LLM-driven evolutionary search with an Edge Transformer evaluator, enabling automated discovery of challenging problem instances and improvements in model generalization without manual benchmark creation.

The paper addresses a fundamental bottleneck in AI research: the lack of systematic methods for evaluating whether neural models truly generalize beyond their training data. Traditional benchmarking relies on human intuition to design harder test cases, which is both labor-intensive and potentially biased. This work leverages LLMs as autonomous benchmark designers, using evolutionary algorithms to discover sampling functions that produce genuinely challenging instances.

The technical contribution spans multiple layers. The researchers use Datalog-based worlds as their problem domain, treating benchmark generation as an optimization problem solvable through FunSearch and agentic search paradigms. By having LLMs propose both problem instances and entirely new reasoning worlds, they create a feedback loop where neural models improve on harder data, and harder data is systematically discovered rather than manually crafted.

This approach has broader implications for AI development. The ability to automatically generate meaningful benchmarks could accelerate research velocity across domains where evaluation has been a bottleneck—from logical reasoning to planning problems. It suggests a future where research progress relies less on human ingenuity in test design and more on collaborative human-AI systems that continuously discover new evaluation frontiers.

The work opens questions about what constitutes genuine hardness in reasoning tasks and whether LLM-discovered challenges align with human intuitions about difficulty. If the methodology generalizes beyond relational reasoning, it could reshape how the AI community validates model capabilities across multiple domains.

• →LLMs can automate benchmark generation by discovering increasingly difficult problem instances through evolutionary search
• →The Edge Transformer improved generalization when trained on LLM-discovered hard instances rather than standard benchmarks
• →The framework enables autonomous exploration of novel reasoning domains, reducing manual benchmark design overhead
• →Automated benchmarking could accelerate AI research by systematically testing generalization beyond training distribution
• →The approach combines evolutionary algorithms with agentic LLM behavior to create self-improving evaluation systems