Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents

Researchers introduce Hierarchical Preference Learning (HPL), a new framework that improves AI agent training by using preference signals at multiple granularities - trajectory, group, and step levels. The method addresses limitations in existing Direct Preference Optimization approaches and demonstrates superior performance on challenging agent benchmarks through a dual-layer curriculum learning system.

• →HPL solves the granularity mismatch problem in training LLM agents by combining trajectory-level, group-level, and step-level preference optimization.
• →The framework decomposes expert trajectories into semantically coherent action groups for more precise credit assignment than traditional methods.
• →A dual-layer curriculum scheduler organizes learning from simple to complex tasks based on group length and sample difficulty.
• →Experimental results show HPL outperforms existing state-of-the-art methods on three challenging agent benchmarks.
• →The approach enables agents to solve both simple behaviors and complex multi-step sequences more effectively.