WRAP++ is a new pretraining technique that enhances language model training by discovering cross-document relationships through web hyperlinks and synthesizing multi-document question-answer pairs. By amplifying ~8.4B tokens into 80B tokens of relational QA data, the method enables models like OLMo to achieve significant performance improvements on factual retrieval tasks compared to single-document approaches.
WRAP++ addresses a fundamental limitation in current LLM pretraining: synthetic data generation has primarily focused on isolated documents, missing the relational knowledge that emerges from connections between sources. The technique leverages web hyperlink structures to identify high-confidence relationships between document pairs—dual-links and co-mentions—then generates question-answer examples requiring reasoning across both documents. This approach creates multiple entry points to the same facts and produces knowledge absent from individual sources alone.
The method builds on the established success of synthetic data rephrasing for LLM training, but extends it into cross-document territory. Traditional single-document rewriting constrains the model's associative context and fails to capture how information relates across sources—a critical aspect of real-world knowledge comprehension. WRAP++ exploits the exponential growth in valid entity pair combinations to achieve a ten-fold data amplification: 8.4B raw tokens expand to 80B tokens of training material.
Performance validation on SimpleQA demonstrates sustained scaling benefits at both 7B and 32B model scales, suggesting the approach generalizes across model sizes. This has direct implications for developers building knowledge-intensive applications requiring factual accuracy and reasoning capabilities. The combinatorial amplification mechanism also addresses a persistent challenge in synthetic data: maintaining sufficient diversity and scale without proportional compute costs.
Future developments will likely explore whether this discovery-driven synthesis applies to non-hyperlinked corpora or domain-specific datasets, and whether the relational knowledge gains persist in downstream applications beyond factual retrieval tasks.
- →WRAP++ discovers cross-document relationships via web hyperlinks to synthesize multi-document QA pairs, amplifying training data 10-fold beyond single-document approaches.
- →Models trained with WRAP++ show substantial performance gains on factual retrieval benchmarks with sustained scaling benefits at multiple model sizes.
- →The combinatorial growth of entity pairs enables exponential data amplification without proportional increases in labeling costs or computational overhead.
- →The method captures associative knowledge patterns that exist only in relationships between documents, creating richer training signals than isolated document rewriting.
- →Wikipedia-scale experiments demonstrate practical viability, converting 8.4B tokens of raw text into 80B tokens of relational training data.