SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics

SPECTRA is a new framework for generating synthetic text corpora and retrieval test collections at scale, enabling researchers to stress-test information retrieval systems without expensive human annotation. The system can produce corpora up to 60,000 documents while maintaining controllable vocabulary distributions and deterministic relevance labels, serving as a diagnostic complement to traditional evaluation methods.

SPECTRA addresses a fundamental bottleneck in information retrieval research: the expense and impracticality of creating large-scale human-judged test collections. Traditional Cranfield and TREC-style evaluation requires extensive manual annotation, which becomes prohibitively costly for proprietary datasets or documents under active development. This framework offers a reproducible alternative by decomposing corpus generation into discrete components—topical structure, text realization, metadata controls, and relevance oracles—that researchers can configure independently.

The technical achievement demonstrates practical scalability, generating corpora at 12,000-14,000 documents per second while preserving realistic linguistic properties like Zipf-distributed vocabulary. The simulation results reveal actionable insights: cross-topic distractor text significantly impacts retrieval performance, with BM25 nDCG@10 dropping from perfect scores to 0.43 as distractor proportion increases to 36%. This diagnostic capability helps identify failure modes before investing in expensive human annotation campaigns.

For the AI research community, SPECTRA reduces barriers to rigorous evaluation of retrieval systems. Teams can now validate scaling assumptions, test architectural changes, and benchmark algorithmic improvements on synthetic data before committing resources to large-scale human studies. This democratizes access to evaluation infrastructure traditionally available only to well-funded institutions. The framework's deterministic nature also ensures reproducibility, a critical requirement for scientific validation that human-annotated collections cannot always guarantee due to inter-annotator variation.

Looking forward, synthetic evaluation frameworks like SPECTRA may reshape how researchers prioritize human annotation efforts, reserving expensive judgments for validating results discovered through cheaper synthetic testing. The technique's applicability extends beyond traditional IR to domain-specific retrieval tasks where private data restrictions prevent conventional evaluation approaches.

• →SPECTRA generates synthetic IR test corpora up to 60,000 documents at 12-14K docs/second, enabling large-scale retrieval system evaluation without human annotation costs.
• →The framework preserves realistic linguistic properties, maintaining Zipf-distributed vocabulary slopes near 0.86 across different corpus sizes.
• →Controlled distractor injection reveals system robustness: BM25 performance degrades from 1.00 to 0.43 nDCG@10 as cross-topic noise increases from 2% to 36%.
• →SPECTRA serves as a diagnostic complement to human evaluation, not a replacement, helping identify failure modes before expensive annotation campaigns.
• →The reproducible, deterministic approach enables systematic testing of retrieval scaling assumptions and architectural decisions with minimal resource investment.