LLM-Guided Reinforcement Learning for Audio-Visual Speech Enhancement

Researchers have developed a new audio-visual speech enhancement framework that uses Large Language Models and reinforcement learning to improve speech quality. The method outperforms existing baselines by using LLM-generated natural language feedback as rewards for model training, providing more interpretable optimization compared to traditional scalar metrics.

• →New AVSE framework combines LLMs with reinforcement learning for better speech enhancement quality.
• →LLM-generated natural language descriptions provide more interpretable feedback than traditional scalar metrics like SI-SNR and MSE.
• →The method uses sentiment analysis to convert LLM descriptions into numerical reward scores for PPO training.
• →Experimental results show superior performance across multiple metrics including PESQ, STOI, and subjective listening tests.
• →The approach addresses the poor correlation between existing metrics and actual perceptual speech quality.