#transformer-circuits News & Analysis

2 articles tagged with #transformer-circuits. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 97/10

🧠

A retrieval conditioned rebinding circuit for dynamic entity tracking in large language models

Researchers have identified a specific neural mechanism in large language models that enables dynamic entity tracking and attribute binding. Using causal analysis, they discovered a retrieval-conditioned rebinding circuit—a compact attention head mechanism that updates entity-attribute relationships as context changes, with distinct architectural implementations across Gemma and Llama model families.

🧠 Llama

AINeutralarXiv – CS AI · Jun 26/10

🧠

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures

Researchers tracked how attention-head circuits form during training across three 1B-parameter language models, revealing that induction circuits and attention-sink circuits emerge as separate phenomena separated by an order of magnitude in training tokens. The study identifies architectural properties (zero BOS-heads in early layers) and demonstrates that circuit identification requires only 0.3-2% of total training data, offering insights into mechanistic interpretability of transformer models.