Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance

Researchers propose HRLLI, a hierarchical reinforcement learning framework that dynamically selects relevant natural-language instruction segments to guide agent decision-making at different stages of task execution. The approach outperforms existing instruction-conditioned RL baselines by treating language as adaptive, stage-specific guidance rather than static input, improving sample efficiency in complex environments.

This research addresses a fundamental limitation in reinforcement learning: poor sample efficiency when training agents to perform complex tasks. Traditional RL systems require millions of environment interactions to learn effective policies, making real-world deployment costly and time-consuming. Recent work has attempted to leverage natural language instructions to accelerate learning, but existing approaches treat instructions as monolithic inputs without considering how different guidance becomes relevant at different task stages.

HRLLI introduces a two-tier policy architecture that fundamentally rethinks how language guidance operates in RL systems. Rather than conditioning the entire agent on static instructions, a high-level semantic policy dynamically selects which instruction fragment applies to the current situation, while a low-level policy executes actions based on this selected guidance. This decomposition mirrors how humans often process complex instructions—breaking them into relevant steps and applying them sequentially based on context.

The framework's practical value emerges from its performance on the RTFM (Read the Fucking Manual) benchmark, a challenging environment specifically designed to test instruction-following capabilities. By explicitly modeling adaptive instruction selection, HRLLI achieves superior sample efficiency compared to baselines, suggesting meaningful improvements in how AI agents can leverage human-provided guidance.

This work carries implications for AI development broadly. As language models become more sophisticated, the ability to integrate natural language guidance into learning systems becomes increasingly important. Better instruction grounding could accelerate training for robotics, autonomous systems, and complex decision-making tasks where environment interactions are expensive or risky. The research validates hierarchical decomposition as a promising direction for sample-efficient learning.

• →HRLLI decomposes natural language instructions into stage-specific guidance elements rather than treating them as monolithic inputs
• →A Select-to-Act paradigm uses high-level semantic policy to select relevant instruction pieces and low-level policy to execute actions
• →The approach demonstrates consistent improvements over instruction-conditioned RL baselines on complex benchmark tasks
• →Dynamic instruction grounding enables agents to adapt language guidance based on current environmental state and task progress
• →Framework improves sample efficiency in RL by leveraging external knowledge more effectively through hierarchical decomposition