Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering

Justitia is a new scheduling system for task-parallel LLM agents that optimizes GPU server performance through selective resource allocation based on completion order prediction. The system uses memory-centric cost quantification and virtual-time fair queuing to achieve both efficiency and fairness in LLM serving environments.

• →Justitia addresses scheduling challenges for task-parallel LLM agents running on shared GPU servers.
• →The system uses memory-centric cost quantification since memory is typically the bottleneck in LLM serving.
• →It employs a lightweight prediction method to estimate agent completion costs accurately.
• →Virtual-time based fair queuing algorithm ensures both performance optimization and worst-case delay guarantees.
• →Implementation on vLLM shows substantial scheduling efficiency improvements while maintaining fairness.