#inference-performance News & Analysis

2 articles tagged with #inference-performance. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBullisharXiv – CS AI · Jun 127/10

🧠

Arbor: Tree Search as a Cognition Layer for Autonomous Agents

Arbor introduces a multi-agent framework using tree search as a cognition layer for autonomous agents operating in complex action spaces. The system achieves 193% inference throughput-latency improvements over vendor baselines through coordinated Orchestrator and Critic agents, demonstrating reproducible, hardware-agnostic optimization across multiple hardware generations.

AIBullisharXiv – CS AI · May 97/10

🧠

When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

Researchers demonstrate that int4 quantization of KV caches on Apple Silicon's unified memory architecture actually improves performance over fp16, delivering 3-8% faster inference while reducing memory usage by 3x. This inverts the traditional quality-latency tradeoff through a fused Metal kernel combining sign-randomized FFT, per-channel scaling, and int4 packing, with applications from 1B to 1.5B parameter models.

🏢 Hugging Face