4 articles tagged with #inference-latency. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Zyphra released Zamba2-VL, a suite of vision-language models combining Mamba2 state-space layers with transformer blocks, achieving competitive performance with leading VLMs while delivering 10x faster time-to-first-token speeds. The three released models (1.2B, 2.7B, 7B parameters) represent a significant efficiency breakthrough for edge and on-device deployment.
Researchers introduce AtlasKV, a parametric knowledge integration method that enables large language models to leverage billion-scale knowledge graphs while consuming less than 20GB of VRAM. Unlike traditional retrieval-augmented generation (RAG) approaches, AtlasKV integrates knowledge directly into LLM parameters without requiring external retrievers or extended context windows, reducing inference latency and computational overhead.
Researchers introduce STAR Benchmark, a new evaluation framework for testing Large Language Models in competitive, real-time environments. The study reveals a strategy-execution gap where reasoning-heavy models excel in turn-based settings but struggle in real-time scenarios due to inference latency.
Researchers developed a novel learning approach for spiking neural networks that optimizes both synaptic weights and intrinsic neuronal parameters, achieving up to 13.50 percentage point improvements in classification accuracy. The study introduces a biologically-inspired SNN-LZC classifier that achieves 99.50% accuracy with sub-millisecond inference latency.
