#low-rank News & Analysis

2 articles tagged with #low-rank. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · 3d ago7/10

🧠

Efficient Pre-Training of LLMs through Truncated SVD Layers

Researchers introduce TSVD, a framework for training Large Language Models more efficiently by maintaining low-rank representations and strict weight orthonormality throughout pretraining. The method uses adaptive rank selection and caching mechanisms to reduce computational overhead while matching or exceeding the performance of standard full-parameter models.

AIBullisharXiv – CS AI · Mar 26/1018

🧠

Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation

Researchers introduce LoRA-Pre, a memory-efficient optimizer that reduces memory overhead in training large language models by using low-rank approximation of momentum states. The method achieves superior performance on Llama models from 60M to 1B parameters while using only 1/8 the rank of baseline methods.