Applied Explainability for Large Language Models: A Comparative Study

Researchers compare three explainability techniques—Integrated Gradients, Attention Rollout, and SHAP—for interpreting LLM decisions on sentiment classification tasks. The study reveals that gradient-based methods offer stability and interpretability, while attention-based approaches are faster but less predictive, highlighting critical trade-offs in choosing explanation methods for transformer models.

This preprint addresses a fundamental challenge in deploying large language models: understanding why they make specific predictions. As LLMs become increasingly integrated into critical applications—from content moderation to financial analysis—the ability to explain model behavior transforms from academic curiosity into practical necessity. Regulators and enterprise stakeholders demand interpretability for compliance and risk management, making this comparative work timely and relevant.

The study's focus on practical evaluation rather than novel methodology reflects a maturation in the explainability field. By systematically comparing three established techniques on a single benchmark task, the researchers provide grounded guidance for practitioners choosing between methods with different computational and interpretability profiles. The finding that gradient-based attribution outperforms attention-based approaches challenges assumptions that attention weights directly correspond to model reasoning—a misconception common in applied settings.

For the AI development community, these results highlight that explainability involves inherent trade-offs. Gradient-based methods demand higher computational resources but yield more reliable insights into prediction mechanisms. Attention-based techniques offer efficiency but may mislead practitioners about which features actually drove decisions. Model-agnostic approaches like SHAP provide generality at the cost of instability and expense.

The implications extend across sectors deploying transformer models. Data scientists and engineers must carefully select explainability tools based on their specific use case requirements—prioritizing either computational efficiency or interpretative confidence. As regulatory frameworks increasingly mandate AI explainability, understanding these method trade-offs becomes essential for responsible deployment. Future work should examine how these findings transfer to larger models and more complex tasks beyond sentiment classification.

• →Gradient-based attribution methods provide more stable and prediction-aligned explanations than attention-based approaches in transformer models.
• →Attention weights do not reliably indicate which features influenced model predictions, challenging common interpretability assumptions.
• →Model-agnostic methods like SHAP offer flexibility but introduce higher computational costs and variance compared to gradient-based techniques.
• →Explainability methods function as diagnostic tools with distinct trade-offs rather than definitive truth sources about model reasoning.
• →Practitioners must select explainability techniques based on specific requirements: computational efficiency versus interpretative reliability.