EdgeCIM: A Hardware-Software Co-Design for CIM-Based Acceleration of Small Language Models

EdgeCIM presents a specialized hardware-software framework designed to accelerate Small Language Model inference on edge devices by addressing memory-bandwidth bottlenecks inherent in autoregressive decoding. The system achieves significant performance and energy improvements over existing mobile accelerators, reaching 7.3x higher throughput than NVIDIA Orin Nano on 1B-parameter models.

EdgeCIM addresses a critical inefficiency in current edge AI infrastructure: while GPUs excel at parallel prefill operations, the sequential token-generation phase relies heavily on memory-bound GEMV computations that underutilize hardware and drain battery life. This research introduces a Computing-in-Memory (CIM) macro implemented at 65nm process technology paired with intelligent tile-based mapping to extract parallelism from inherently sequential workloads. The framework enables meaningful performance gains—achieving 336 tokens per second and 173 tokens per joule under INT4 precision across multiple model families.

The development reflects growing recognition that general-purpose accelerators cannot efficiently handle the distinct computational patterns of language model inference stages. Mobile processors like Snapdragon and edge GPUs like Orin Nano struggle with decoding workloads precisely because they optimize for throughput-bound operations rather than latency-critical memory patterns. EdgeCIM's specialized approach represents a broader industry trend toward domain-specific architectures for AI inference.

For edge computing stakeholders—smartphone manufacturers, embedded systems developers, and edge AI platform providers—this work demonstrates viable pathways to real-time language model inference without cloud dependencies. The 49.59x energy efficiency improvement over Orin Nano has practical implications for battery-constrained devices and cost-intensive IoT deployments. The extensive benchmarking across diverse model architectures (LLaMA, Phi, Qwen, SmolLM) validates generalizability rather than single-model optimization.

Future developments will likely focus on manufacturing feasibility at scale, software integration with existing inference frameworks, and exploration of quantization-aware design trade-offs. Success hinges on whether these theoretical advantages translate to commercial silicon, particularly given the capital requirements for chip fabrication.

• →EdgeCIM achieves 7.3x throughput improvement over NVIDIA Orin Nano and 49.59x better energy efficiency on small language models through specialized CIM architecture
• →The framework addresses the memory-bandwidth bottleneck in autoregressive decoding, the dominant bottleneck in decoder-only model inference on edge devices
• →Performance validated across eight different model families and configurations, demonstrating generalization beyond single-model optimization
• →INT4 quantized inference delivers 336 tokens/second and 173 tokens/joule on edge hardware, enabling practical real-time applications
• →Domain-specific accelerators for AI inference represent a divergence from general-purpose GPU-centric approaches, with implications for silicon design strategy