Contribution-aware Token Compression for Efficient Video Understanding via Reinforcement Learning

Researchers developed CaCoVID, a reinforcement learning-based algorithm that compresses video tokens for large language models by selecting tokens based on their actual contribution to correct predictions rather than attention scores. The method uses combinatorial policy optimization to reduce computational overhead while maintaining video understanding performance.

• →CaCoVID addresses the computational overhead problem in video large language models caused by redundant video tokens.
• →The algorithm uses reinforcement learning to optimize token selection based on contribution to correct predictions rather than attention scores.
• →A combinatorial policy optimization algorithm reduces exploration space and accelerates convergence in token selection.
• →The method shifts from passive token preservation to active discovery of optimal compressed token combinations.
• →Extensive experiments on video understanding benchmarks demonstrate the effectiveness of the compression approach.