GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval

GIRL-DETR introduces a novel reinforcement learning approach for video moment retrieval that addresses the optimization gap between training losses and evaluation metrics. By freezing backbone networks and applying progressive RL only to detection heads, the method achieves significant accuracy improvements while protecting learned feature representations in lightweight models.

Video moment retrieval—pinpointing specific temporal segments in videos based on text descriptions—represents a critical computer vision challenge with applications ranging from video search to content moderation. Traditional approaches suffer from a fundamental optimization problem: the losses used during training don't directly align with the metrics used for evaluation, causing models to plateau and settle into suboptimal solutions. The GIRL-DETR framework addresses this through architectural and training innovations that separate feature learning from metric optimization.

The research builds on recent trends in applying reinforcement learning to vision tasks, particularly post-training optimization for large models. Prior work demonstrated RL's effectiveness for metric-aware training but noted that directly applying RL to lightweight networks degrades performance by disrupting fragile learned representations. GIRL-DETR solves this through gradient isolation—freezing the backbone and query generation systems while allowing only the detection head to receive RL updates. This orthogonal decoupling prevents optimization interference.

The technical contributions include Cross-Modal Interaction for early text-video alignment and Text-Guided Gating to inject semantic information into transformer queries before prediction. The Three-stage Progressive RL strategy gradually increases optimization complexity, avoiding dramatic distribution shifts. Validation across three benchmark datasets (Charades-STA, QVHighlights, TACoS) demonstrates substantial accuracy gains with minimal parameter updates.

For the AI research community, this work establishes a practical pathway for applying RL to resource-constrained models, potentially enabling metric-optimized training beyond large-scale systems. The approach's efficiency gains matter for deployment scenarios prioritizing inference speed or edge computing, expanding RL's applicability beyond high-capacity networks.

• →GIRL-DETR decouples feature learning and metric optimization through gradient isolation, preventing RL from disrupting supervised pre-training
• →The method achieves significant accuracy improvements on video moment retrieval benchmarks while updating only detection head parameters
• →Cross-Modal Interaction and Text-Guided Gating mechanisms enhance query quality before transformer decoding
• →Three-stage Progressive RL strategy prevents distribution shift while gradually increasing optimization complexity
• →Approach demonstrates RL's effectiveness for lightweight models, expanding beyond large-scale systems