#real-world-performance News & Analysis

3 articles tagged with #real-world-performance. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBullisharXiv – CS AI · Jun 127/10

🧠

Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System

Researchers developed a pre-response classifier for clinical LLMs that predicts user rejection risk with 71.9% accuracy by leveraging deployment-specific context like provider type and department. This deployment-centered evaluation approach addresses a critical gap in clinical AI assessment, moving beyond static benchmarks to measure real-world user acceptance in a healthcare system.

AIBearisharXiv – CS AI · Jun 106/10

🧠

Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?

Researchers benchmarked 7 frontier LLMs against China's National Computer Rank Examination, a standardized office proficiency test with 200 practical tasks across Word, Excel, and PowerPoint. Single-turn models achieved only 36.6% accuracy, while advanced agentic systems with iterative feedback reached 68.8%, revealing significant gaps in LLM-based office automation despite recent code-generation improvements.

AIBearisharXiv – CS AI · Mar 276/10

🧠

Back to Basics: Revisiting ASR in the Age of Voice Agents

Researchers introduced WildASR, a multilingual diagnostic benchmark revealing that current ASR systems suffer severe performance degradation in real-world conditions despite achieving near-human accuracy on curated tests. The study found that ASR models often hallucinate plausible but unspoken content under degraded inputs, creating safety risks for voice agents.