SCALE: Scalable Cross-Attention Learning with Extrapolation for Agentic Workflow Scheduling

Researchers introduce SCALE, a deep reinforcement learning scheduler that enables LLM-based agentic systems to generalize across different cluster sizes without retraining. Using cross-attention architecture and a novel regularization technique, the system achieves 8.9% improvement in response times when scaled from 16 to 48 nodes, addressing a critical infrastructure challenge for distributed AI workloads.

SCALE addresses a fundamental inefficiency in current AI infrastructure: existing schedulers must be completely retrained whenever computational clusters change size. This constraint creates significant operational friction for enterprises deploying agentic LLM systems that decompose complex tasks into workflow graphs requiring careful resource allocation across heterogeneous hardware.

The innovation combines two technical approaches. The cross-attention pointer network architecture accepts variable server counts by design, allowing tasks to query against dynamic server pools. However, architectural flexibility alone proves insufficient—the researchers discovered that attention features undergo distribution shift as cluster size increases, degrading performance at unseen scales. Their solution, Structured Representation Regularization (SRR), uses decorrelation loss and KL penalties to maintain stable feature statistics regardless of input size.

For infrastructure operators and AI service providers, this research directly impacts deployment costs and operational complexity. Current systems require expensive retraining cycles when scaling infrastructure, a particular problem as enterprises expand AI deployments. SCALE's ability to generalize without fine-tuning could reduce infrastructure management overhead and enable more dynamic resource allocation.

The 8.9% improvement in average response time at 48 nodes demonstrates practical value, though real-world impact depends on how response time improvements translate to user experience and infrastructure utilization in production environments. Future work should validate performance on truly heterogeneous clusters with diverse hardware types, as the current evaluation assumes uniform node configurations. The research also doesn't address the initial training cost or performance degradation limits when scaling far beyond training scale.

• →SCALE enables LLM schedulers to generalize across cluster sizes without retraining, reducing operational overhead for AI infrastructure teams.
• →Structured Representation Regularization (SRR) prevents attention feature distribution shift, the key bottleneck preventing simple architecture scaling.
• →8.9% response time improvement at 48 nodes demonstrates practical efficiency gains in distributed agentic systems.
• →The architecture accepts any number of servers by construction, making infrastructure scaling more flexible and cost-efficient.
• →Results suggest explicit regularization is necessary for neural networks to maintain performance across different input scales.