#model-consistency News & Analysis

3 articles tagged with #model-consistency. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBearisharXiv – CS AI · 3d ago7/10

🧠

How Reliable Are AI Attackers Against a Fixed Vulnerable Target? A 400-Run Empirical Study of LLM Penetration Testing Consistency

Researchers conducted 400 autonomous penetration testing runs across four LLM models against a fixed vulnerable target to measure attack consistency. Results show significant variation in exploitation success rates (25-85%) and distinctive failure modes per model, with Claude and Gemini 2.5 Flash-Lite substantially outperforming GPT-4o-mini and Qwen, raising critical questions about LLM reliability in security-critical autonomous operations.

🏢 Anthropic🧠 GPT-4🧠 Claude

AINeutralarXiv – CS AI · Apr 147/10

🧠

Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling

Researchers introduce Accelerated Prompt Stress Testing (APST), a new evaluation framework that reveals safety vulnerabilities in large language models through repeated prompt sampling rather than traditional broad benchmarks. The study finds that models appearing equally safe in conventional testing show significant reliability differences when repeatedly queried, indicating current safety benchmarks may mask operational risks in deployed systems.

AINeutralarXiv – CS AI · May 76/10

🧠

How Does Thinking Mode Change LLM Moral Judgments? A Controlled Instant-vs-Thinking Comparison Across Five Frontier Models

Researchers compared moral judgment consistency in five frontier LLMs when using instant versus extended reasoning modes across 100 scenarios. While overall agreement remained statistically similar between modes, reasoning improved cross-model consensus on disputed moral cases and reduced demographic-based inconsistencies, suggesting that explicit reasoning processes may enhance fairness despite not dramatically shifting individual verdicts.

🧠 GPT-5🧠 Claude🧠 Sonnet