Explaining Black-Box Language Models: Learning to Optimize Linguistically-Structured Word Subsets

Researchers propose a novel method for explaining black-box language model predictions by identifying linguistically-structured word subsets without requiring access to internal model parameters or gradients. The approach uses reinforcement learning and graph-based linguistic knowledge to generate interpretable, efficient explanations that outperform existing methods across multiple architectures and datasets.

This research addresses a critical gap in AI interpretability: explaining decisions from black-box language models deployed in high-stakes environments like healthcare. Traditional explanation methods struggle with three competing constraints—inference efficiency, avoiding out-of-distribution artifacts, and linguistic coherence. The proposed solution formulates explanation as an amortized optimization problem, eliminating the need for expensive input-specific searches at inference time.

The technical innovation lies in combining REINFORCE policy gradients with graph-structured linguistic knowledge. This hybrid approach enables discrete word selection without direct gradient access to the black box, while anchoring explanations in linguistic structure rather than raw statistical patterns. By respecting grammatical and syntactic relationships, the method produces explanations aligned with human intuition rather than opaque numerical scores.

For practitioners deploying large language models through APIs, this work directly addresses transparency requirements increasingly mandated by regulators in healthcare, finance, and other regulated sectors. The method's efficiency enables real-time explanations without significant latency overhead. Evaluation across diverse architectures demonstrates robustness, suggesting the approach generalizes beyond specific model families.

The implications extend beyond compliance. Interpretable AI builds user trust and enables detection of spurious correlations or biases the model may exploit. As organizations face mounting pressure to demonstrate algorithmic fairness and accountability, efficient black-box explanation methods become competitive advantages. Future development likely focuses on scaling these techniques to even larger models and integrating explanations into user-facing applications where cognitive load matters.

• →The method enables efficient, black-box compatible explanations without access to model parameters or gradients, addressing deployment constraints for API-based models.
• →Linguistic structure integration produces explanations grounded in grammar and syntax, improving alignment with human interpretation versus purely statistical approaches.
• →REINFORCE-based policy gradients enable discrete word selection in a fully gradient-free setting, eliminating the need for input-specific optimization at inference time.
• →Evaluation demonstrates stronger performance than gradient-based approaches with oracle model access, suggesting the trade-offs favor practical applicability.
• →The work supports growing regulatory requirements for AI transparency in high-stakes domains like healthcare and finance.