Why the Valuable Capabilities of LLMs Are Precisely the Unexplainable Ones

A research paper argues that the most valuable capabilities of large language models are precisely those that cannot be captured by human-readable rules. The thesis is supported by proof showing that if LLM capabilities could be fully rule-encoded, they would be equivalent to expert systems, which have been proven historically weaker than LLMs.

• →The paper proposes that LLMs' most valuable capabilities are inherently unexplainable through discrete human-readable rules.
• →A proof by contradiction demonstrates that rule-encodable LLM capabilities would be equivalent to expert systems, which are historically weaker.
• →The research draws on the Chinese philosophical concept of Wu (sudden insight through practice) to support the thesis.
• →The findings have significant implications for AI interpretability research and safety approaches.
• →There exists a structural mismatch between human cognitive tools and the complexity of modern AI systems.