EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors

Researchers introduce EPSVec, a differentially-private method for generating synthetic data using large language models that operates significantly more efficiently than existing approaches. By using dataset vectors to steer LLM generation, the technique decouples privacy costs from the number of synthetic samples generated, enabling high-quality synthetic data creation even with limited private datasets.

EPSVec addresses a critical bottleneck in machine learning: the scarcity of shareable high-quality training data. Organizations sitting on sensitive datasets—financial records, medical data, proprietary corpora—face a dilemma: keep data locked away or risk privacy violations by sharing it. Synthetic data generation via LLMs offers a middle path, but existing private generation methods suffer from severe inefficiencies, requiring substantial computational resources and large batch sizes to maintain usable quality.

The innovation here centers on dataset vectors, mathematical directions in neural activation space that capture what makes private data statistically distinct from public training corpora. Rather than repeatedly applying privacy-preserving operations during generation, EPSVec extracts these vectors once and sanitizes them through differential privacy mechanisms upfront. Subsequent generation requires no additional privacy budget, enabling unlimited synthetic samples from a single privacy investment.

For machine learning practitioners and enterprises, this represents a meaningful improvement in the privacy-utility tradeoff. Organizations can now generate abundant synthetic training data from limited sensitive corpora without multiplicative computational costs or privacy risks scaling with sample count. The method shows particular promise in low-data regimes where traditional approaches struggle—exactly where many organizations find themselves.

The broader implications extend to regulated industries where data sharing remains restricted. Financial institutions, healthcare providers, and government agencies could accelerate model development using synthetic training data derived from real patterns without exposing sensitive information. Future work likely involves integration with commercial ML platforms and validation across additional sensitive data domains beyond text.

• →EPSVec decouples privacy budget from generation volume, eliminating the computational penalty of creating many synthetic samples
• →Dataset vectors capture statistical differences between private and public data, enabling efficient steering of LLM generation
• →The method demonstrates superior performance in low-data regimes where existing synthetic generation approaches typically fail
• →Single upfront extraction and sanitization of steering vectors reduces overall computational overhead significantly
• →Strong distributional alignment and downstream utility improvements make synthetic data suitable for practical machine learning applications