CSR: Infinite-Horizon Real-Time Policies with Massive Cached State Representations

Researchers introduce Cached State Representation (CSR), a framework that reduces latency in deploying large language models for robotics by 26-fold through optimized token caching and asynchronous state management. The approach enables real-time robot control with massive language models while maintaining full contextual understanding over infinite operational horizons.

The deployment of large language models in robotics faces a critical engineering bottleneck: the time required to process extensive state histories before generating the first token (TTFT). This latency problem makes real-time robot control impractical with existing approaches. The CSR framework addresses this by formalizing optimal computational structures around three theoretical principles—prefix stability, incremental extensibility, and asynchronous reconciliation—which together enable maximum key-value cache reuse without sacrificing context.

This work builds on years of research attempting to balance context windows against inference speed. Previous solutions either sacrificed global context through windowing techniques or created prohibitive computational overhead. The Asynchronous State Reconciliation algorithm represents the practical innovation that sustains these properties continuously by offloading memory management to parallel computational resources, eliminating the latency spikes that previously plagued long-horizon tasks.

The empirical results are substantial: on a wirelessly-connected robot with a 235-billion parameter model, the framework achieved 26-fold latency reduction (14.67 seconds reduced to 0.56 seconds) while processing 120,000-token contexts. Beyond raw speed, the system achieved state-of-the-art recall metrics (0.836 versus 0.459) on embodied AI benchmarks, demonstrating that performance gains didn't compromise reasoning quality.

This advancement directly enables a class of applications previously infeasible: high-frequency (over 2 Hz) continuous robot policies powered by frontier language models. The implications extend beyond robotics to any embodied AI system requiring real-time decision-making with extensive contextual reasoning. Future work likely focuses on scaling to even larger models and multi-robot coordination scenarios.

• →CSR achieves 26-fold latency reduction for large language model inference in robotics through optimized KV-cache reuse.
• →Asynchronous State Reconciliation algorithm maintains real-time performance over infinite operational horizons without latency spikes.
• →Framework enables high-frequency (>2 Hz) robot control policies using 235B parameter models with 120K token contexts.
• →Achieves state-of-the-art recall (0.836) on embodied AI benchmarks while maintaining production-grade latency requirements.
• →Solves fundamental engineering bottleneck in deploying frontier language models for continuous real-world robotic systems.