#throughput-improvement News & Analysis

3 articles tagged with #throughput-improvement. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBullisharXiv – CS AI · Jun 107/10

🧠

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

Researchers present a CPU-GPU hybrid system enabling local deployment of large Mixture-of-Experts models with cloud-level performance, achieving 1,800 tokens/s throughput and supporting 45K-token prompts within 30 seconds using consumer hardware. The breakthrough addresses critical gaps in local inference including latency, throughput, and concurrent workload handling without requiring quantization or model distillation.

AIBullisharXiv – CS AI · May 17/10

🧠

Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed

Researchers introduce Efficient-DLM, a framework for converting pretrained autoregressive language models into diffusion language models that enable parallel, non-autoregressive generation. The approach uses block-wise attention patterns and position-dependent masking to preserve model accuracy while achieving 4.5x higher throughput compared to existing models.

AIBullisharXiv – CS AI · Feb 277/106

🧠

veScale-FSDP: Flexible and High-Performance FSDP at Scale

Researchers introduce veScale-FSDP, a redesigned Fully Sharded Data Parallel system that overcomes limitations of current FSDP implementations used for training large-scale AI models. The new system features flexible RaggedShard format and structure-aware planning, achieving 5-66% higher throughput and 16-30% lower memory usage while supporting advanced training methods and scaling to tens of thousands of GPUs.