#parameter-space News & Analysis

2 articles tagged with #parameter-space. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AINeutralarXiv – CS AI · Jun 106/10

🧠

Recoverable but Not Stationary:Local Linear Structures in Weights and Activations

Researchers demonstrate that linear structures in neural networks exist locally rather than globally, with task-specific directions that evolve during training rather than remaining stationary. Their findings on transformer models and LoRA adapters suggest that parameter adjustment techniques like task vectors work through dynamic geometric patterns that partially align across weight and activation spaces.

AINeutralarXiv – CS AI · Jun 86/10

🧠

On the Geometry of On-Policy Distillation

Researchers characterize the training dynamics of on-policy distillation (OPD), a technique used to improve large language model reasoning, revealing it operates in a distinct geometric regime compared to supervised fine-tuning and reinforcement learning. The study shows OPD exhibits 'subspace locking,' where cumulative updates rapidly converge to a narrow low-dimensional channel that is functionally sufficient for performance, suggesting OPD has unique training dynamics rather than existing as a simple intermediate between other training approaches.