Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning

Researchers demonstrate that large language models can be effectively fine-tuned to perform sequential decision-making tasks across MDPs, POMDPs, and ambiguous environments by learning from offline trajectory data. The approach achieves stronger performance than baseline methods, particularly in complex, partially-observed scenarios, with theoretical analysis showing the fine-tuned attention mechanisms implicitly estimate optimal Q-functions.

This research bridges two previously siloed domains: large language models and reinforcement learning-style sequential decision-making. The work reveals that LLMs possess latent capabilities for planning and policy learning that can be unlocked through supervised fine-tuning on offline data rather than requiring explicit RL training algorithms. This matters because it suggests a simpler, more practical pathway to deploy LLMs in real-world decision-making contexts.

The theoretical contribution is particularly significant. By interpreting fine-tuned attention layers as implicit Q-function estimators, the authors provide formal grounding for why this approach works, deriving suboptimality bounds that separate in-context estimation error from training-length bias. This theoretical clarity distinguishes the work from pure empirical demonstrations and enables future optimization.

The practical implications are substantial for domains with abundant offline data but sparse online interaction opportunities, notably healthcare and finance. Rather than collecting interactive trajectory data or deploying complex RL algorithms, practitioners can leverage existing datasets to fine-tune pretrained LLMs for decision-making tasks. The consistent improvements across synthetic environments—especially in long-horizon and partially-observed settings—demonstrate robustness rather than narrow applicability.

Looking forward, the critical questions center on scaling to real-world datasets and action spaces, transferability across domains, and computational efficiency during deployment. Integration with existing LLM infrastructure and compatibility with prompt-based adaptation methods will determine practical adoption. The intersection of offline learning, LLMs, and sequential decision-making represents a growing frontier where similar work will likely accelerate.

• →Fine-tuned LLMs achieve substantially smaller optimality gaps than baseline methods in sequential decision-making tasks across MDPs, POMDPs, and ambiguous environments.
• →Theoretical analysis shows fine-tuned attention layers implicitly estimate optimal Q-functions, providing formal grounding for the approach's effectiveness.
• →The method is particularly advantageous for domains with abundant offline data but limited online interaction, such as healthcare and finance.
• →Performance gains are especially pronounced in long-horizon, partially-observed, and model-ambiguous settings compared to in-context-only approaches.
• →Supervised fine-tuning offers a practical alternative to traditional reinforcement learning algorithms for endowing LLMs with decision-making capabilities.