Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep

Researchers introduce HetCache, a training-free acceleration framework for diffusion-based video editing that achieves 2.67x speedup by selectively caching contextually relevant tokens instead of processing all attention operations. The method reduces computational redundancy in Diffusion Transformers while maintaining video editing quality and consistency.

• →HetCache framework achieves 2.67x latency speedup for diffusion-based video editing without training requirements.
• →The method addresses architectural redundancy in Diffusion Transformers by selectively caching context tokens based on correlation strength.
• →Approach divides spatial-temporal tokens into context and generative categories, focusing computation on most relevant interactions.
• →Significant FLOPs reduction achieved with negligible degradation in video editing quality and consistency.
• →Framework specifically targets masked video-to-video generation and editing applications.