PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Researchers introduced PIPE-Cypher, an automated pipeline for generating Text-to-Cypher benchmarks tailored to enterprise property graphs. The system combines schema profiling, LLM generation, and validation to create deployment-relevant datasets that reflect real user queries, addressing the challenge that enterprise graphs have unique structures and evolving schemas that make standardized benchmarks inadequate.

PIPE-Cypher addresses a critical gap in AI evaluation infrastructure for enterprise systems. Traditional benchmarks fail to capture the diversity of enterprise knowledge graphs, which vary significantly in schema design, domain terminology, and governance requirements. The research demonstrates that generic Text-to-Cypher models struggle with zero-shot transfer to new domains, indicating that benchmarking practices must become deployment-specific rather than one-size-fits-all.

The technical approach is comprehensive, incorporating schema profiling to understand graph structure, reverse-query grounding to validate queries against actual entities, and local LLM judging to maintain consistency without cloud dependencies. By successfully generating 3,000 balanced benchmark examples from live production graphs, the authors prove that automated benchmark creation at scale is feasible. The ablation studies confirm that model performance improves substantially with schema-specific few-shot examples, validating the core hypothesis that context-aware training data matters more than raw model capability.

This work has implications for enterprise adoption of AI-powered query systems. Organizations deploying Text2Cypher solutions can now establish repeatable benchmarking processes that evolve with their infrastructure, enabling continuous evaluation as schemas change. The emphasis on local generation and judging addresses enterprise privacy concerns around exposing proprietary graph data to cloud services. The research also signals that successful AI systems in enterprise contexts require domain-specific optimization rather than relying on general-purpose models trained on public datasets.

• →PIPE-Cypher automates generation of enterprise-specific Text-to-Cypher benchmarks that reflect actual user query patterns and graph schemas.
• →The pipeline achieves 3,000 validated examples using local Qwen 3.5-9B models, demonstrating that scalable benchmark creation doesn't require large cloud infrastructure.
• →Zero-shot transfer performance remains weak across models, confirming that generic benchmarks fail to capture domain-specific complexity.
• →Schema-specific few-shot learning significantly improves model performance, validating that enterprise AI deployment requires contextual fine-tuning.
• →The repeatable, evolving benchmark process enables continuous evaluation as enterprise graphs and user workloads change over time.