Safety-Aware Evaluation of LLM-Generated Driver Intervention Messages through Multi-Task Risk Fusion

Researchers propose the Driver Safety-Aware Intervention Score (DSAIS), a domain-specific metric for evaluating LLM-generated driver safety messages across five dimensions including risk-urgency alignment and cognitive load. The framework integrates multi-task recognition outputs through risk fusion and achieves strong inter-rater reliability (ICC 0.798-0.840), demonstrating that compact local LLMs outperform API-based models for in-vehicle deployment.

This research addresses a critical gap in autonomous vehicle safety systems by moving beyond generic text evaluation metrics like BLEU and BERTScore to domain-specific assessment. Driver intervention messages require nuanced evaluation across dimensions that generic metrics cannot capture—particularly the alignment between message urgency and actual risk levels, cognitive load on drivers, and driver acceptability. DSAIS represents a meaningful advancement in safety-critical AI evaluation methodology.

The work builds on growing recognition that LLM outputs in safety-critical applications demand specialized evaluation frameworks. Traditional metrics optimize for linguistic similarity rather than functional effectiveness in life-or-death scenarios. By combining lightweight rule-based computation with LLM Judge evaluation, the researchers created a hybrid approach that balances computational efficiency with semantic understanding—crucial for real-time vehicle systems.

The experimental findings carry significant implications for automotive AI deployment. The 9.1% improvement in contextual relevance through multi-task integration (emotion recognition, hazard detection, etc.) demonstrates that holistic context matters substantially. More practically, the discovery that compact 7B-9B parameter local models outperform larger API-based alternatives addresses real deployment constraints in vehicles with limited computational resources and connectivity requirements.

The identification of driver emotion recognition as the most critical upstream factor fundamentally shapes how future in-vehicle systems should be architected. Rather than optimizing for message clarity alone, systems must account for driver psychological state. This finding extends beyond academic interest—it directly informs production vehicle design decisions and regulatory frameworks for autonomous systems. The strong inter-rater reliability metrics (ICC 0.798-0.840) validate DSAIS as a deployable evaluation standard for the industry.

• →DSAIS metric achieves ICC 0.798-0.840 reliability for evaluating safety intervention messages across five contextual dimensions.
• →Multi-task integration improves contextual relevance by 9.1% over rule-based systems, showing fusion architecture benefits.
• →Compact local LLMs (7B-9B parameters) outperform larger API-based models, enabling practical in-vehicle deployment without cloud dependency.
• →Driver emotion recognition emerges as the most critical upstream factor for intervention message effectiveness.
• →Hybrid rule-based and LLM Judge architecture balances computational efficiency with semantic evaluation for safety-critical applications.