Memory Caching: RNNs with Growing Memory

Researchers introduce Memory Caching (MC), a technique that enhances recurrent neural networks by allowing their memory capacity to grow with sequence length, bridging the gap between fixed-memory RNNs and growing-memory Transformers. The approach offers four variants and shows competitive performance with Transformers on language modeling and long-context tasks while maintaining better computational efficiency.

• →Memory Caching allows RNNs to have growing memory capacity that scales with sequence length, similar to Transformers but with better efficiency.
• →The technique offers a flexible trade-off between RNNs' O(L) complexity and Transformers' O(L²) complexity.
• →Four MC variants are proposed, including gated aggregation and sparse selective mechanisms.
• →Experimental results show MC-enhanced recurrent models perform competitively with Transformers on recall-intensive tasks.
• →The approach addresses a key limitation of recurrent architectures in sequence modeling applications.