The Missing Memory Hierarchy: Demand Paging for LLM Context Windows

Researchers developed Pichay, a demand paging system that treats LLM context windows like computer memory with hierarchical caching. The system reduces context consumption by up to 93% in production by evicting stale content and managing memory more efficiently, addressing fundamental scalability issues in AI systems.

• →Current LLM context windows waste 21.8% of tokens on structural overhead like stale tool results and system prompts.
• →Pichay implements a memory hierarchy for LLMs with L1 cache, L2 fault-driven pinning, and L3 conversation compaction.
• →The system achieved up to 93% reduction in context consumption while maintaining 99.97% fault-free operation in production.
• →LLM context limits and attention degradation are fundamentally virtual memory problems that can be solved with established computer science techniques.
• →Three levels of the memory hierarchy are already deployed in production, with cross-session memory identified as the next frontier.