SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer

Researchers developed Score Matched Actor-Critic (SMAC), a new offline reinforcement learning method that enables smooth transition to online RL algorithms without performance drops. SMAC achieved successful transfer in all 6 D4RL tasks tested and reduced regret by 34-58% in 4 of 6 environments compared to best baselines.

• →SMAC solves the common problem of performance drops when transitioning offline RL models to online fine-tuning.
• →The method regularizes Q-functions during offline training to maintain derivative equality between policy scores and action-gradients.
• →SMAC achieved smooth transfer to Soft Actor-Critic and TD3 algorithms across all tested D4RL benchmark tasks.
• →The approach reduces regret by 34-58% compared to existing methods in two-thirds of tested environments.
• →The research provides evidence that offline and online RL maxima are separated by low-performance valleys in loss landscapes.