A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Models

Researchers apply game-theoretic free energy principles to analyze attention head interactions in large language models, discovering that heads exhibit higher-order redundancy. Their framework enables principled pruning of low-contribution heads, achieving 18% FLOP reduction and 22% throughput improvement in GPT2 with minimal performance degradation.

This research bridges game theory and deep learning by treating transformer attention heads as bounded rational agents optimizing a shared objective. The Game Theoretic Free Energy Principle (GTFEP) framework decomposes multi-head interactions into interpretable components—pairwise mutual information and higher-order interaction information—revealing how heads coordinate across different scales. The discovery of consistently negative triple dividends across BERT, GPT2, and Llama indicates that attention mechanisms develop redundant representations, where three or more heads collectively provide less unique information than their pairwise combinations suggest.

The practical implications are substantial for model efficiency. By identifying heads with marginal contributions to overall performance, researchers can prune them without degradation proportional to their removal. The demonstrated results—maintaining near-baseline perplexity while reducing computational cost—address a critical challenge in deploying large models. Current LLMs face memory and latency constraints that limit real-world adoption; efficient architectures directly reduce infrastructure costs and enable deployment on resource-constrained devices.

This work establishes theoretical foundations for understanding why transformers work despite their apparent over-parameterization. Rather than treating pruning as an empirical exercise, GTFEP provides principled metrics for identifying redundancy. The Nash equilibrium correspondence ensures that pruned configurations remain stable operating points, not degraded approximations. Future applications could extend this framework to dynamic pruning during inference, adaptive computation based on input complexity, or architecture search guided by free energy principles. The convergence of game theory and interpretability opens pathways for designing more efficient and understandable neural architectures.

• →Game theoretic analysis reveals attention heads exhibit negative higher-order interactions, indicating systematic redundancy in transformer architectures
• →Principled head pruning achieves 18% FLOP reduction and 22% throughput gains with minimal perplexity increase across tested models
• →GTFEP framework provides theoretical guarantees that pruned configurations remain Nash equilibria, validating efficiency improvements
• →Higher-order interaction information metrics enable identification of marginally-contributing heads without retraining
• →Framework applies consistently across BERT, GPT2, and Llama, suggesting universal redundancy patterns in transformer attention