AINeutralarXiv – CS AI · 3h ago6/10
🧠
The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
A comprehensive systematic review of 337 studies examines how Transformer-based language models encode syntactic knowledge, finding strong performance on formal syntax but variable results at the syntax-semantics interface. The research reveals that while these models demonstrate non-trivial syntactic abilities through behavioral and mechanistic evidence, understanding the detailed computational mechanisms remains limited due to methodological heterogeneity and heavy concentration on English and BERT-like architectures.