Shape and Substance: Dual-Layer Side-Channel Attacks on Local Vision-Language Models

Researchers discovered significant privacy vulnerabilities in local Vision-Language Models that use Dynamic High-Resolution preprocessing. The dual-layer attack framework can exploit execution-time variations and cache patterns to infer sensitive information about processed images, even when models run locally for privacy.

• →Dynamic High-Resolution preprocessing in VLMs creates algorithmic side-channels that leak information about input geometry and content type.
• →Attackers can use unprivileged OS metrics to fingerprint image aspect ratios through execution-time analysis.
• →Cache contention profiling enables distinguishing between visually dense and sparse content within identical image geometries.
• →Popular models like LLaVA-NeXT and Qwen2-VL are vulnerable to these privacy inference attacks.
• →Proposed security mitigations involve substantial performance overhead, creating challenging trade-offs for Edge AI deployments.