Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

Researchers introduce AFTER, a benchmark evaluating how procedural memory in large language models transfers across tasks, roles, and model types. Testing on 382 enterprise tasks across six professional roles, the study finds that procedural memory improves performance by 3.7-6.7 points per refinement round, with multi-model trained skills achieving 73.1% cross-model accuracy—though some skills generalize broadly while others become role-specific.

The development of procedural memory systems for LLM agents addresses a critical gap in enterprise AI deployment: understanding whether learned skills can be reused efficiently across different workplace contexts. AFTER's comprehensive benchmark represents significant progress in moving beyond theoretical AI research toward practical production systems that organizations depend on daily. The 382-task evaluation spanning multiple professional roles and skill categories provides empirical grounding often missing in AI research.

Procedural memory has emerged as organizations seek to optimize LLM agent performance on repetitive, complex workflows. Rather than retraining models from scratch for each new task, procedural memory allows agents to encode and reuse learned procedures. This approach mirrors how human workers develop expertise through practice and procedural learning. The benchmark's structured evaluation framework—testing local improvement, cross-task transfer, cross-role transfer, and cross-model generalization—reveals nuances about skill transferability that matter for real-world deployment.

The findings carry substantial implications for enterprises building AI agent platforms. A 3.7-6.7 point performance improvement per refinement cycle directly translates to cost savings and operational efficiency gains. The 73.1% cross-model accuracy for multi-model trained skills suggests organizations don't need to retrain procedures when upgrading model backbones, reducing migration friction. However, the discovery that some skills specialize to specific roles while others generalize widely indicates practitioners must carefully evaluate transferability before scaling procedures across their organizations.

Future development hinges on understanding which task characteristics enable broad generalization versus role-specific specialization. Organizations deploying procedural memory systems should focus on identifying high-value, broadly transferable procedures while developing separate skill libraries for role-specific workflows, optimizing both performance and maintenance complexity.

• →Procedural memory improves LLM agent performance by 3.7-6.7 points per refinement round in enterprise workflows.
• →Skills trained across multiple models achieve 73.1% cross-model test accuracy, enabling seamless model upgrades without retraining.
• →Procedural memory effectiveness varies significantly: some skills transfer broadly across tasks and roles while others specialize and lose effectiveness when transferred.
• →The AFTER benchmark's 382-task evaluation provides empirical guidance for building production-ready procedural memory systems.
• →Cross-model generalization outperforms single-model trace sources, indicating diverse training data improves skill robustness.