Learning Question-Aware Keyframe Selection with Synthetic Supervision for Video Question Answering

Researchers developed a question-aware keyframe selection framework for video question answering that uses large multimodal models to generate pseudo labels and coverage regularization. The method significantly improves accuracy on temporal and causal questions in the NExT-QA dataset, making video analysis more efficient by reducing inference costs.

• →Large multimodal models show promise in video question answering but face challenges with high inference costs and information dilution.
• →The new framework uses pseudo keyframe labels from LMMs combined with coverage regularization to select diverse, complementary video frames.
• →Experiments on NExT-QA dataset demonstrate significant accuracy improvements, particularly for temporal and causal question types.
• →Keyframe selection is established as an effective and learnable module that can enhance VideoQA efficiency.
• →The approach addresses limitations of relying solely on image-text similarity for frame selection in video analysis.