When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop

A new study reveals that human curation efforts to align AI models can backfire in multi-model ecosystems where models train on outputs from other models. While curation improves alignment in isolated systems, cross-model interactions can dampen or reverse these benefits, potentially degrading long-term alignment across interconnected AI systems.

This research addresses a critical vulnerability in how modern AI systems are developed and deployed. As foundation models increasingly rely on synthetic data from prior iterations rather than purely human-generated content, a self-consuming loop emerges that risks model collapse or bias amplification. Prior work demonstrated that human curation could mitigate these risks in single-model scenarios, suggesting a straightforward path to safer AI development.

However, the real-world deployment landscape differs fundamentally from isolated lab conditions. In practice, AI systems interact with outputs from competing or complementary models, creating complex feedback networks. This paper formalizes that dynamic and reveals an inconvenient truth: human curation of one model can inadvertently contaminate or misalign other models that consume its outputs. The effect propagates asymmetrically through the system, meaning investments in curating one model may yield diminishing returns or even harm alignment elsewhere.

For AI developers and organizations building production systems, this finding challenges assumptions about scalable alignment strategies. Curation costs remain high, and this research suggests those costs scale poorly across multi-model architectures without careful system design. The implication is that alignment cannot be solved model-by-model but requires ecosystem-level coordination and governance.

Looking forward, this work opens questions about optimal curation strategies in interconnected systems. Future research may explore how to design model interactions that preserve alignment benefits while preventing cross-model contamination. Organizations deploying multiple models should anticipate that siloed curation efforts may be insufficient and consider how data flows between systems affect overall safety properties.

• →Human curation improves AI alignment in isolated models but can degrade it across multi-model ecosystems through unintended cross-influences.
• →Self-consuming training loops where models learn from prior iterations create alignment risks that scale unpredictably with system complexity.
• →Current model-by-model curation strategies may fail to achieve alignment goals when models interact and share synthetic data flows.
• →The research formalizes conditions for stable convergence in multi-model systems, providing a framework for predicting alignment outcomes.
• →Effective AI safety in production may require ecosystem-level governance rather than isolated per-model alignment efforts.