Discovering Interpretable Algorithms by Decompiling Transformers to RASP

Researchers present a method to extract interpretable programs from trained Transformers by converting them to RASP (a simple programming language) and using causal interventions to identify minimal sub-programs. Experiments on algorithmic tasks demonstrate that length-generalizing Transformers often implement simple, understandable algorithms internally, providing direct evidence that neural networks discover human-readable solutions.

This research addresses a fundamental question in deep learning interpretability: whether neural networks that generalize well actually implement the simple algorithms we expect them to find. The study bridges transformer architecture and formal computation theory by leveraging prior work showing Transformers can be expressed in RASP, a minimal programming language designed to study algorithmic reasoning. The key innovation is a practical method that converts trained models into RASP equivalents, then uses causal interventions to strip away unnecessary complexity and isolate core algorithmic logic.

The work builds on growing momentum in mechanistic interpretability—the field focused on understanding internal neural network reasoning. Previous research established the theoretical link between RASP expressiveness and Transformer length generalization, but demonstrating that actual trained models discover such clean implementations remained an open question. This paper closes that gap empirically.

For the AI research community, these findings validate assumptions about how neural networks solve problems and strengthen arguments for scaling interpretability techniques. The methodology could accelerate understanding of other model families and task domains. However, the current experiments remain limited to small Transformers on algorithmic tasks, leaving questions about scalability to modern large language models and real-world applications.

The broader implications suggest neural networks may be more interpretable than previously assumed when incentive structures reward simple solutions. As organizations increasingly require explainable AI systems, techniques for extracting human-readable algorithms from trained models become practically valuable for auditing and verification purposes.

• →Researchers developed a method to extract interpretable RASP programs from trained Transformers using re-parameterization and causal interventions.
• →Experiments confirm that length-generalizing Transformers on algorithmic tasks internally implement simple, human-readable algorithms.
• →The approach bridges neural network learning and formal computation theory, validating theoretical predictions about how models solve problems.
• →Current results are limited to small models on algorithmic tasks, with scalability to larger models and complex domains remaining unclear.
• →This work advances mechanistic interpretability, enabling better understanding and auditing of neural network decision-making processes.