Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding

Researchers introduce LyraV, a streaming video-language model that maintains real-time synchronization between video perception and language generation without pausing. The system uses a hierarchical control framework with two key components—a Frame-Driven Transition Controller and Streaming Token Pacer—to interleave video frames with generated tokens at 3.89 FPS with 98.29% synchrony.

LyraV addresses a fundamental limitation in current Video-LLMs: the inability to process video and generate responses simultaneously in real-time streaming contexts. Traditional architectures pause video perception during token generation, creating stutters and breaking the natural flow of interaction. This research introduces a paradigm shift toward Streaming Video-Language Synchrony (SVLS) that enables truly continuous, frame-aware understanding without compromising response quality.

The technical innovation centers on two components working in tandem. The Frame-Driven Transition Controller operates as a verification-based finite-state machine that makes semantic decisions about when to continue speaking, initiate new responses, or remain silent—without requiring additional training. The Streaming Token Pacer dynamically adjusts generation speed to match visual content pace, deploying only small token chunks per frame interval to maintain the real-time budget. This per-frame incremental decoding approach fundamentally changes how video-understanding systems can operate.

For the AI development community, this work demonstrates that streaming video understanding is not merely an engineering problem but requires rethinking model architecture itself. The results across five online and three offline benchmarks show LyraV preserves general understanding while dramatically improving streaming performance. The emergent capability for dynamic reasoning over streaming tokens—enabling continuous interpretation alongside visual input—suggests these models may develop novel cognitive patterns when freed from batch processing constraints.

Looking forward, this approach could influence how future video-AI systems are designed, particularly for live streaming applications, real-time commentary, and interactive video analysis. The technique's training-free nature makes it potentially applicable to existing Video-LLM architectures, suggesting rapid adoption pathways.

• →LyraV eliminates video perception pauses during response generation, achieving 98.29% synchrony with video playback.
• →The Frame-Driven Transition Controller operates as a training-free verification system for semantic decision-making.
• →Streaming Token Pacer dynamically adjusts language generation rate to match visual content pace per-frame.
• →Model preserves general understanding ability while substantially improving streaming fluency and narrative coherence.
• →Emergent capability for dynamic reasoning over streaming tokens enables continuous interpretation alongside visual input.