AIBearisharXiv – CS AI · May 286/10
🧠A research study reveals that NPUs (Neural Processing Units) on mobile devices don't consistently accelerate LLM inference as expected, with CPUs outperforming NPUs on compute-intensive prefill operations and NPUs providing only marginal speedups on memory-bound decode stages. The findings challenge assumptions about heterogeneous mobile computing and suggest current NPU designs require architectural improvements for on-device AI workloads.
AINeutralarXiv – CS AI · May 286/10
🧠Researchers propose SelfJudge, a new method for accelerating large language model inference through self-supervised judge verification that eliminates the need for human annotations. The approach trains verifiers to assess whether token substitutions preserve semantic meaning, enabling faster inference without sacrificing accuracy across diverse NLP tasks.
AINeutralarXiv – CS AI · May 276/10
🧠Researchers propose a RAG-based framework leveraging Large Language Models to detect and mitigate Carpet-Bombing DDoS attacks in Software-Defined Networks. The system achieves high detection accuracy without traditional supervised training, addressing a critical vulnerability in SDN's centralized architecture through intelligent traffic behavior classification.
AIBullisharXiv – CS AI · May 276/10
🧠Researchers propose PIPO (Pair-In, Pair-Out), a novel technique that combines input compression and multi-token prediction to accelerate large language model inference. The method eliminates expensive verification steps while achieving up to 2.64x speedups in first-token latency and demonstrating significant improvements on reasoning benchmarks.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers present a diagnostic framework for evaluating KV cache eviction selectors in large language models, identifying three failure modes and demonstrating that value-aware ranking combined with evidence recovery achieves 72.6% accuracy on positive-margin test cases. The work addresses a critical bottleneck in long-context LLM inference by revealing why compression strategies succeed or fail.
AIBullisharXiv – CS AI · May 126/10
🧠Researchers present KV-RM, a runtime optimization that manages KV-cache memory movement in static-graph LLM decoders, achieving better throughput and reduced latency variability without sacrificing the predictability benefits of static graph execution. The approach decouples logical KV histories from physical storage through a block pager and merge-staged transport mechanism, demonstrating practical improvements on multi-GPU systems.
🏢 Nvidia
AIBullisharXiv – CS AI · May 96/10
🧠Researchers propose a reinforcement learning-based policy for routing intermediate reasoning steps across language models of varying sizes, reducing inference costs while maintaining accuracy on math benchmarks. The method uses threshold calibration to balance performance and efficiency without requiring large process reward models, outperforming handcrafted routing strategies.
AINeutralarXiv – CS AI · May 46/10
🧠A technical study comparing Nvidia and Apple Silicon for running large language models locally reveals fundamental architectural trade-offs: Nvidia achieves higher throughput through specialized quantization but faces memory constraints requiring aggressive model compression, while Apple's unified memory architecture scales more efficiently with superior energy performance. The research highlights ecosystem fragmentation as a major barrier for consumer adoption of datacenter-scale AI inference.
🏢 Nvidia
AINeutralarXiv – CS AI · May 16/10
🧠Researchers introduce PAD-Rec, a lightweight module that optimizes speculative decoding for LLM-based recommendation systems by incorporating position-aware embeddings. The approach achieves up to 3.1x speedup in inference while preserving recommendation quality, addressing the latency bottleneck in generative list-wise recommendations.
AINeutralarXiv – CS AI · Apr 146/10
🧠ConfigSpec introduces a profiling-based framework for optimizing distributed LLM inference across edge-cloud systems using speculative decoding. The research reveals that no single configuration can simultaneously optimize throughput, cost efficiency, and energy efficiency—requiring dynamic, device-aware configuration selection rather than fixed deployments.
AINeutralarXiv – CS AI · Apr 146/10
🧠A-IO addresses critical memory-bound bottlenecks in LLM deployment on NPU platforms like Ascend 910B by tackling the 'Model Scaling Paradox' and limitations of current speculative decoding techniques. The research reveals that static single-model deployment strategies and kernel synchronization overhead significantly constrain inference performance on heterogeneous accelerators.
AIBullisharXiv – CS AI · Mar 176/10
🧠Researchers propose Outcome-Aware Tool Selection (OATS), a method to improve tool selection in LLM inference gateways by interpolating tool embeddings toward successful query centroids without adding latency. The approach improves tool selection accuracy on benchmarks while maintaining single-digit millisecond CPU processing times.
AIBullisharXiv – CS AI · Mar 37/106
🧠Researchers have developed AloePri, the first privacy-preserving LLM inference method designed for industrial applications. The system uses collaborative obfuscation to protect input/output data while maintaining 96.5-100% accuracy and resisting state-of-the-art attacks, successfully tested on a 671B parameter model.
