#tensor-parallelism News & Analysis

3 articles tagged with #tensor-parallelism. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBullisharXiv – CS AI · May 287/10

🧠

GQLA: Group-Query Latent Attention for Hardware-Adaptive Large Language Model Decoding

Researchers propose Group-Query Latent Attention (GQLA), an advancement of DeepSeek's Multi-head Latent Attention that enables hardware-adaptive decoding through two algebraically equivalent inference paths without requiring model retraining. The innovation allows a single trained model to optimize performance across different hardware platforms—H100 GPUs and export-restricted H20 chips—while maintaining computational efficiency and supporting distributed tensor parallelism.

AIBullisharXiv – CS AI · Jun 96/10

🧠

Scaling Neural Network Verification with Tensor Parallelism and Fully Sharded Data Parallelism

Researchers have adapted GPU parallelism techniques to neural network verification, enabling formal safety proofs on larger models. Fully Sharded Data Parallelism (FSDP) reduces memory usage by 80-90% while maintaining identical verification results, though Tensor Parallelism trades some bound quality for memory efficiency.

$COMP

AINeutralarXiv – CS AI · May 116/10

🧠

CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training

Researchers introduce CommFuse, a novel communication-computation overlap technique that eliminates tail latency in distributed LLM training by decomposing collective operations into peer-to-peer communications. The method improves efficiency for both tensor parallelism and data parallelism across GPU/TPU/NPU clusters, achieving higher throughput and model FLOPS utilization compared to existing solutions.