Researchers propose a new theoretical framework for understanding visual text compression (VTC) using measure transport theory, which reveals that token savings don't reliably predict performance gains. They develop label-free methods to identify when visual encoding helps or hurts performance, achieving 70% accuracy in matching oracle decisions and improving average task scores by 3.3% while reducing tokens by 10.3%.
Visual text compression represents a promising but unpredictable approach to handling long-context language processing by converting text to images and processing them through vision-language models. While VTC can achieve 3-20x token reduction compared to traditional subword tokenization, the research community has lacked principled methods to predict when this efficiency translates into actual performance gains. This paper addresses a fundamental gap: compression metrics alone cannot determine task utility, leaving practitioners unable to reliably decide when to apply VTC.
The measure transport framework provides the missing theoretical lens by modeling text and visual tokens as probability distributions. The researchers decompose the information loss into precision costs (from within-patch aggregation) and coverage costs (from cross-patch fragmentation), both measurable without labeled data. This theoretical grounding enables practical innovations: a label-free routing mechanism that selects between visual and text paths per input, and a foveation technique that adaptively increases resolution in high-cost regions.
The empirical validation across 24 NLP datasets demonstrates meaningful real-world applicability. Achieving 70.8% oracle-matching accuracy on routing decisions suggests the framework captures genuine task-relevant properties. The simultaneous achievement of better average performance (+3.3%) with fewer tokens (-10.3%) indicates this isn't a false efficiency trade-off but genuine improvement in the visual encoding pipeline.
For the broader AI community, this work bridges theory and practice in multimodal processing. Developers can now apply VTC more confidently without ground truth labels, enabling deployment in resource-constrained environments. The transport-theoretic approach may inspire similar principled frameworks for other compression and multimodal fusion problems, establishing standards for evaluating information loss in neural systems.
- →Visual text compression achieves 3-20x token reduction but lacks principled methods to predict when performance gains occur.
- →Measure transport theory decomposes information loss into precision and coverage costs, both measurable without labeled data.
- →A label-free routing mechanism matches oracle decisions on 70.8% of tasks, enabling practical deployment decisions.
- →Adaptive foveation re-encodes high-cost regions at higher resolution, improving average task scores by 3.3% with 10.3% fewer tokens.
- →The theoretical framework provides foundations for understanding multimodal compression beyond visual text encoding.