One for All: A Non-Linear Transformer can Enable Cross-Domain Generalization for In-Context Reinforcement Learning

Researchers propose a non-linear transformer architecture that enables reinforcement learning agents to generalize across different domains through in-context learning, establishing a theoretical connection between transformers and kernel-based temporal difference learning. By interpreting transformers as operators in Reproducing Kernel Hilbert Space, the work demonstrates that value functions from diverse domains can share a unified weight set, with MetaWorld experiments validating the approach.

This research addresses a fundamental challenge in reinforcement learning: enabling models trained on specific tasks to perform effectively on entirely new domains without retraining. Rather than relying on traditional multi-task or meta-RL approaches, the authors leverage transformer architectures' natural ability to adapt through in-context learning—similar to how large language models generalize across topics. The key innovation lies in reinterpreting transformers through a kernel-based mathematical lens, connecting them to temporal difference learning algorithms that have long underpinned RL.

The theoretical framework treats transformers as functional operators mapping context sequences to task-specific value functions within a Reproducing Kernel Hilbert Space. This perspective provides mathematical rigor for understanding why transformers enable cross-domain generalization: when value functions from different RL domains inhabit the same RKHS, a shared set of weights can represent all of them simultaneously. This unifies previously disparate learning approaches.

For the broader AI and machine learning community, this work bridges theoretical understanding and practical performance. The MetaWorld experimental validation demonstrates that the theory translates to real algorithmic improvements. This carries implications for developing more robust RL systems capable of deployment in diverse environments—from robotics to autonomous systems—without expensive domain-specific retraining cycles. The approach potentially reduces the computational and data requirements for achieving generalization across related tasks.

Future research should explore scaling these insights to more complex domain variations and investigating whether the RKHS framework extends to other architectural choices beyond transformers. Understanding the conditions under which domains share an RKHS remains crucial for practical applications.

• →Non-linear transformers enable reinforcement learning agents to generalize across different domains via in-context learning without explicit parameter retraining.
• →The work establishes a theoretical connection between transformers and kernel-based temporal difference learning through Reproducing Kernel Hilbert Space interpretation.
• →Shared weight representations become possible when value functions from different domains exist within the same RKHS mathematical space.
• →MetaWorld experiments validate that the theoretical framework produces convergent temporal-difference objectives across multiple domains.
• →This research has implications for building more generalizable RL systems in robotics and autonomous systems with reduced retraining overhead.