Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories

Researchers found that LLM-generated stories suffer from severe lack of diversity, with just 11 specific words appearing in 88.3% of outputs across multiple models. These recurring elements—character names like Elias and Mara, settings like lighthouses, and professions like clockmaker—originate from preference data used in model alignment rather than training data, revealing how small datasets can disproportionately shape AI outputs.

The research exposes a critical vulnerability in how large language models are aligned and fine-tuned. While LLMs train on vast amounts of internet data, the preference datasets used for alignment—typically curated to remove harmful content—are comparatively tiny. When these small datasets contain specific examples, alignment algorithms amplify them dramatically, creating artificial patterns that dominate model behavior across different architectures. This phenomenon reveals a fundamental tension in AI development: the very mechanisms designed to make models safer and more controllable can introduce unwanted homogeneity.

The findings carry important implications for AI developers and users alike. For creative applications like storytelling, the lack of diversity suggests current LLMs produce formulaic, predictable content despite their impressive linguistic capabilities. This undermines their utility for writers, game developers, and content creators seeking varied narratives. The research also highlights how alignment practices, while necessary for safety, can have unintended consequences that distort model behavior in unexpected ways.

Looking ahead, this work points toward the need for more diverse and carefully curated preference datasets. Developers must balance safety considerations with output variety, potentially through larger, more representative alignment datasets or refined fine-tuning techniques. The study's findings may prompt renewed discussion about the trade-offs inherent in current alignment approaches, particularly as models become more capable and their outputs more consequential across industries.

• →Just 11 words appear in 88.3% of LLM stories across four different models, indicating systematic homogeneity rather than model-specific behavior.
• →Recurring elements originate from preference data used for alignment, not from training data or published literature, highlighting the outsized influence of small curated datasets.
• →The alignment algorithms meant to improve safety and control inadvertently amplify specific patterns from preference data to disproportionate levels.
• →Current LLM storytelling outputs are far more formulaic than typical post-training content, suggesting alignment removes natural diversity alongside harmful material.
• →The research indicates developers must redesign alignment processes to preserve output variety while maintaining safety guardrails in AI systems.