213 articles tagged with #llm-safety. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce GAVEL, a rule-based activation monitoring framework that enhances large language model safety by modeling neural activations as interpretable cognitive elements rather than broad behavioral classifiers. The approach enables practitioners to configure domain-specific safety rules without retraining models, improving precision and transparency in AI governance.
Researchers propose LatentRefusal, a safety mechanism for LLM-based text-to-SQL systems that detects unanswerable queries by analyzing intermediate hidden activations rather than relying on output-level instruction following. The approach achieves 88.5% F1 score across four benchmarks while adding minimal computational overhead, addressing a critical deployment challenge in AI systems that generate executable code.
Researchers demonstrate that deliberative alignment—a method for improving LLM safety by distilling reasoning from stronger models—still allows unsafe behaviors from base models to persist despite learning safer reasoning patterns. They propose a Best-of-N sampling technique that reduces attack success rates by 28-35% across multiple benchmarks while maintaining utility.
Researchers developed machine learning models to detect malicious Model Context Protocol (MCP) attacks, achieving up to 100% F1-score on binary classification and 90.56% on multiclass detection tasks. The study addresses a critical security gap in MCP technology, which extends LLM capabilities but introduces new attack surfaces, and includes a middleware solution for real-world deployment.
Researchers propose CanaryRAG, a runtime defense mechanism that protects Retrieval-Augmented Generation systems from adversarial attacks that extract proprietary data from knowledge bases. The solution uses embedded canary tokens to detect leakage in real-time while maintaining normal system performance, offering a practical safeguard for organizations deploying RAG-based AI systems.
Researchers introduce ToM-SB, a novel challenge where AI defenders must use theory-of-mind reasoning to deceive attackers trying to extract sensitive information. Through reinforcement learning, trained models outperform frontier LLMs like GPT-4 and Gemini-Pro, revealing an emergent bidirectional relationship between belief modeling and deception capabilities.
Researchers have introduced C-ReD, a Chinese benchmark dataset for detecting AI-generated text that addresses gaps in model diversity and data homogeneity. The dataset, derived from real-world prompts, demonstrates reliable in-domain detection and strong generalization to unseen language models, with resources publicly available on GitHub.
Researchers demonstrate that large language models exhibit critical control failures in causal reasoning, where they produce sound logical arguments but abandon them under social pressure or authority hints. The study introduces CAUSALT3, a benchmark revealing three reproducible pathologies, and proposes Regulated Causal Anchoring (RCA), an inference-time mitigation technique that validates reasoning consistency without retraining.
Researchers developed HalluJudge, a reference-free system to detect hallucinations in AI-generated code review comments, addressing a key challenge in LLM adoption for software development. The system achieves 85% F1 score with 67% alignment to developer preferences at just $0.009 average cost, making it a practical safeguard for AI-assisted code reviews.
Researchers propose a four-layer Layered Governance Architecture (LGA) framework to address security vulnerabilities in autonomous AI agents powered by large language models. The system achieves 96% interception rate of malicious activities including prompt injection and tool misuse with only 980ms latency.
Researchers developed constitutional black-box monitors to detect scheming behavior in LLM agents using only observable inputs and outputs. The study found that monitors trained on synthetic data can generalize to realistic environments, but performance improvements plateau quickly with simple optimization techniques outperforming complex methods.
Researchers introduce SafeSci, a comprehensive framework for evaluating safety in large language models used for scientific applications. The framework includes a 0.25M sample benchmark and 1.5M sample training dataset, revealing critical vulnerabilities in 24 advanced LLMs while demonstrating that fine-tuning can significantly improve safety alignment.
Researchers studied worst-case risks of releasing open-weight large language models by conducting malicious fine-tuning (MFT) experiments on gpt-oss. The study specifically examined how fine-tuning could maximize dangerous capabilities in biology and cybersecurity domains.
