3 articles tagged with #large-scale-models. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
LoTUS is a novel machine unlearning method that removes the influence of training data from pre-trained models without requiring full retraining. The approach smooths prediction probabilities to reduce over-confidence from memorized data and introduces a new evaluation metric (RF-JSD) for real-world conditions, outperforming existing methods on large-scale datasets like ImageNet1k.
Researchers developed a training method for large-scale Mixture-of-Experts (MoE) models using FP4 precision on Hopper GPUs without native 4-bit support. The technique achieves 14.8% memory reduction and 12.5% throughput improvement for 671B parameter models by using FP4 for activations while keeping core computations in FP8.
Researchers introduce veScale-FSDP, a redesigned Fully Sharded Data Parallel system that overcomes limitations of current FSDP implementations used for training large-scale AI models. The new system features flexible RaggedShard format and structure-aware planning, achieving 5-66% higher throughput and 16-30% lower memory usage while supporting advanced training methods and scaling to tens of thousands of GPUs.
