Domain-Specific Data Synthesis for LLMs via Minimal Sufficient Representation Learning

Researchers introduce DOMINO, a framework that synthesizes domain-specific training data for large language models by learning from reference examples rather than explicit domain descriptions. The approach combines prompt tuning with contrastive learning to generate diverse, high-quality synthetic data without manual prompt engineering, improving coding task performance by up to 4.63%.

DOMINO addresses a fundamental limitation in LLM fine-tuning: the difficulty of acquiring sufficient high-quality domain-specific training data. Traditional data synthesis methods require explicit domain knowledge articulated through natural language prompts and careful engineering, which is impractical when domain characteristics are implicit or difficult to formalize. This research pivots to an inductive approach where the domain itself is defined through reference examples, enabling practical adaptation without manual specification.

The technical innovation lies in learning what the authors call a 'minimal sufficient representation'—essentially distilling the essential patterns that distinguish a domain from general data. By integrating contrastive disentanglement objectives, DOMINO separates genuine domain-level characteristics from sample-specific noise, addressing a key challenge in synthetic data generation: overfitting to training examples while missing broader domain applicability. The theoretical guarantee that synthetic data support expands ensures generated samples aren't merely memorized variations of inputs.

This work carries significant implications for AI practitioners developing specialized LLM applications. Currently, domain adaptation requires either expensive manual data collection or sophisticated prompt engineering expertise. DOMINO democratizes this process by automating domain characterization from examples alone. The coding benchmark results, where domains are inherently difficult to articulate verbally, demonstrate real-world applicability beyond controlled research settings.

The framework represents progress toward more accessible domain adaptation, potentially reducing barriers for organizations building specialized AI systems in vertical markets where labeled data is scarce but exemplars exist. Future applications likely extend beyond coding to scientific research, financial analysis, and legal document processing.

• →DOMINO learns domain characteristics inductively from reference examples, eliminating reliance on explicit natural language domain descriptions.
• →The framework combines prompt tuning with contrastive disentanglement to separate core domain patterns from sample-specific noise.
• →Theoretical analysis proves DOMINO expands synthetic data distribution support, ensuring greater diversity and generalization.
• →Empirical results on coding benchmarks show up to 4.63% improvement in Pass@1 accuracy over instruction-tuned baselines.
• →This approach enables practical domain adaptation without manual prompt design, reducing barriers for specialized LLM deployment.