98 articles tagged with #model-interpretability. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers demonstrate that toxic language in prompts significantly degrades the factual accuracy of large language models, even when semantic content remains identical. By analyzing internal model activations, they identify that toxicity amplifies perturbation-sensitive nodes while leaving core reasoning pathways relatively stable, revealing a critical vulnerability in LLM reliability.
Researchers propose a framework to attribute AI model behavior to specific development stages (pretraining, fine-tuning, alignment), enabling accountability tracking without model retraining. The method quantifies how each stage contributes to model outputs and can identify spurious correlations, advancing transparency in AI development.
Researchers propose Bottom-up Policy Optimization (BuPO), a novel reinforcement learning approach that optimizes internal layers of language models rather than treating them as unified policies. The study reveals that LLMs contain distinct internal policy structures with different entropy patterns across layers, offering new insights into how transformer-based models process reasoning tasks.
Researchers introduce Temporal Logit Observability (TLO), a training-free diagnostic tool that reveals how LLM jailbreak attacks unfold over time by analyzing logit patterns during decoding, rather than just whether attacks succeed. The method identifies that attacks with identical success rates actually follow different failure pathways, enabling better safety evaluation and early-stopping defenses that reduce successful jailbreaks by over 50%.
Researchers introduce VLA-Trace, a diagnostic framework for analyzing Vision-Language-Action models that reveals how these AI systems transform multimodal inputs into physical control actions. The study identifies that popular VLA models like π₀.₅ and OpenVLA exhibit distinct adaptation patterns, rely on different routing strategies during decision-making, but struggle with fine-grained semantic understanding despite excelling at visual grounding.
Researchers compared five post-hoc explainability methods for interpreting deep learning models trained to detect Major Depressive Disorder from EEG data. While different attribution approaches showed partially overlapping patterns emphasizing frontal and temporal brain regions, the study reveals methodological assumptions significantly influence interpretability results, cautioning against treating findings as definitive clinical biomarkers.
Researchers introduce CB-SLICE, a new method for identifying systematic errors in deep learning models by leveraging Concept Bottleneck Models to detect error patterns linked to human-understandable concepts. The approach outperforms existing techniques in uncovering model biases and provides more accurate, interpretable explanations of failure modes across multiple benchmarks.
Researchers investigated how transformer language models track entity states through multiple changes, finding that LMs use a non-incremental parallel aggregation strategy rather than sequential state tracking. The study reveals LMs implement state removal operations through a fragile global suppression mechanism, explaining various failure modes and suggesting mechanistic improvements for more robust entity tracking.
Researchers present CODE, a novel approach to knowledge editing in large language models that replaces fact overwriting with causal reasoning. By embedding causal narratives and on-policy distillation into model parameters, CODE reduces self-refutation rates from 95.6% to 1.8%, enabling LLMs to evolve knowledge coherently rather than storing isolated facts.
Researchers demonstrate that Vision-Language Models (VLMs) used for optical character recognition produce fluent but visually unsupported text, relying heavily on language priors rather than actual image content. Testing on Ancient Greek critical editions reveals VLMs generate plausible errors while traditional OCR produces local noise, with token-level grounding analysis showing model-specific vulnerabilities to hallucination.
Researchers demonstrate a successful attack on Introspection Adapters, a technique proposed by Shenoy et al., by exploiting symmetry properties in the system. The findings highlight potential vulnerabilities in adapter-based AI architectures that could have implications for model security and trustworthiness.
A new arXiv paper challenges recent claims that large language models can introspect and monitor their own internal states. By re-examining two popular evaluation paradigms, researchers demonstrate that LLM success appears to stem from surface-level pattern matching rather than genuine metacognition, with models failing to distinguish between internal state tampering and input manipulation.
Researchers propose FEPoID, a training-free method for automatically selecting optimal layers in large language models to detect hallucinations. The approach outperforms existing criteria and baselines while introducing a truncation strategy that further enhances detection performance across question answering and summarization tasks.
Researchers investigated why chain-of-thought prompting improves language model accuracy by analyzing what happens at inference time rather than generation time. They discovered that the improvement comes primarily from lexical activation and short-range token co-occurrence (2-3 adjacent tokens) rather than from logical sentence-level reasoning, challenging assumptions about how rationales actually drive model performance.
Researchers propose π-Soft-NC and π-Soft-NS, improved evaluation metrics for assessing input attribution methods in large language models that control for the number of retained words, addressing a fundamental bias in existing faithfulness evaluation frameworks. They also introduce Grad-ELLM, a gradient-based attribution method designed for decoder-only LLMs that combines gradient and attention mechanisms for stronger explanatory performance.
Researchers introduce ROSS, a robust out-of-distribution detection framework that combines median smoothing with instability quantification to defend machine learning systems against adversarial attacks. The method achieves state-of-the-art performance by leveraging the observation that OOD samples exhibit higher instability under perturbations, outperforming prior defenses by up to 40 AUROC points.
Researchers have analyzed how audio-visual large language models (AVLLMs) process cross-modal information, discovering that integrated audio-visual data concentrates in specialized 'sink tokens' rather than distributing uniformly. This finding enables a training-free method to reduce hallucinations by leveraging these cross-modal information hubs.
Researchers have developed a geometric framework for understanding how large language models process information across their layers, identifying three distinct phases in next-token prediction: Seeding Multiplexing, Hoisting Overriding, and Focal Convergence. The study reveals that model depth primarily increases capacity for candidate disambiguation rather than adding fundamentally new computational stages.
Researchers present a Sequential Forward Floating Selection (SFFS) framework for identifying the minimal set of satellite imagery channels needed for accurate landslide detection, demonstrating that 8 carefully selected channels match or exceed the performance of models using 30 channels. The work addresses computational efficiency and model interpretability in Earth observation machine learning by moving beyond conventional approaches that simply include all available data.
Researchers introduce a scalable framework for evaluating large language models using Item Response Theory and majorization-minimization algorithms, achieving orders-of-magnitude speedups while improving interpretability. The method addresses computational limitations of traditional benchmarking approaches and provides insights into model abilities and benchmark item characteristics.
Researchers introduce a proxy-analyzer framework that detects hallucinations in large language models by analyzing internal activations of a small open-weight reader model rather than the generator itself. The system achieves competitive or superior performance compared to existing methods across multiple model architectures, with notably consistent results showing that model size has minimal impact on detection accuracy.
Researchers have developed supervised sparse auto-encoders (SAEs) that improve mechanistic interpretability of neural networks by addressing non-smoothness issues in L1 penalties and aligning learned features with human semantics. Validated on Stable Diffusion 3.5, the method enables compositional generalization and feature-level interventions for semantic image editing without prompt modification.
Researchers have successfully adapted Vision-Language Models (VLMs) based on LLaMA 3.2 to classify neutrino events in high-energy physics detector data, demonstrating that transformer-based architectures outperform traditional CNNs while offering superior interpretability. This work showcases the broader applicability of large multimodal AI models beyond natural language processing to specialized scientific domains.
Researchers introduce FAMPE, a novel attribution method that uses frequency-domain analysis to improve explainability in deep neural networks. By separately perturbing high and low-frequency components through FFT-based techniques, the method outperforms existing attribution approaches on ImageNet across multiple architectures without requiring manual baseline selection.
Researchers introduce PKeX-Shapley, an algorithm that computes exact Shapley values for product-kernel machine learning models in quadratic time, eliminating the need for approximations. The method exploits the multiplicative structure of product kernels to achieve linear-time-per-feature attribution without sampling or density estimation, extending beyond predictive models to statistical discrepancy measures like MMD and HSIC.
