Emotion-Aware Image Generation from Korean Diary Text via LLM-based Prompt Translation and LoRA Fine-Tuning

Researchers propose an emotion-aware text-to-image pipeline that uses large language models and fine-tuned Stable Diffusion to generate children's drawing-style images from Korean diary entries. The system combines sentiment recognition via Qwen3-8B with LoRA-fine-tuned image generation, addressing T2I models' inability to capture emotional context effectively.

This research addresses a fundamental limitation in current text-to-image generation: the gap between semantic understanding and emotional context. While T2I models excel at rendering visual objects and scenes, they struggle with implicit sentiment and nuanced emotional states present in personal narratives like diaries. The proposed pipeline tackles this by decoupling the sentiment recognition task from image generation, using specialized language models for emotional analysis before prompt engineering.

The technical approach reflects broader trends in multimodal AI: combining multiple specialized models rather than relying on single end-to-end systems. Qwen3-8B handles sentiment extraction, while Stable Diffusion 3.5 Medium generates visuals. The LoRA fine-tuning on children's drawings represents a practical application of parameter-efficient adaptation, allowing domain-specific image generation without full model retraining. This modular design has implications for content creation in domains requiring emotional fidelity.

The practical impact extends to therapeutic and educational applications. Emotion-aware image generation could support mental health applications, children's learning tools, or creative writing platforms. The focus on Korean diary text suggests cultural-specific NLP challenges, particularly around implicit emotional expression in non-English languages.

The authors' critique of CLIP Score as an evaluation metric for emotion-aware generation highlights a critical gap: existing metrics prioritize visual-linguistic alignment rather than emotional accuracy. This observation may drive development of emotion-specific evaluation frameworks. Future work should explore cross-cultural applicability and real-world deployment in therapeutic or educational settings.

• →Emotion-aware T2I requires separate sentiment recognition before image generation, not end-to-end models.
• →LoRA fine-tuning enables domain-specific image styles without full model retraining.
• →Standard evaluation metrics like CLIP Score inadequately measure emotional accuracy in generated images.
• →LLM-based prompt translation bridges implicit sentiment in source text to visual representations.
• →Cultural and linguistic specificity (Korean diary analysis) reveals NLP challenges in non-English emotional understanding.