VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

Researchers introduce VisionFoundry, a synthetic data generation pipeline that uses LLMs and text-to-image models to create targeted training data for vision-language models. The approach addresses VLMs' weakness in visual perception tasks and demonstrates 7-10% improvements on benchmark tests without requiring human annotation or reference images.

VisionFoundry tackles a fundamental limitation in how vision-language models are trained: natural image datasets provide insufficient supervision for low-level visual skills like spatial reasoning and viewpoint recognition. The research demonstrates that synthetic, task-targeted data can systematically improve VLM capabilities where real-world datasets fall short. This addresses a critical gap in the development pipeline for AI systems that need robust visual understanding beyond broad pattern matching.

The approach represents an evolution in AI training methodology. Rather than relying exclusively on web-scraped natural images with noisy labels, VisionFoundry leverages the generative capabilities of LLMs and text-to-image models to create precisely annotated synthetic datasets. The pipeline generates questions, answers, and training images from only a task keyword, then validates consistency using a proprietary VLM—eliminating manual annotation overhead. This scalability aspect differentiates it from traditional dataset construction approaches.

For the AI development community, these results suggest synthetic supervision can systematically address model bottlenecks without massive labeling efforts. The 7% improvement on MMVP and 10% on CV-Bench-3D benchmarks, combined with favorable scaling properties, indicate this isn't a narrow-case optimization but a generalizable training approach. Developers building multimodal systems can reference this methodology to identify and remediate specific capability gaps.

The significance extends beyond benchmark improvements. If synthetic task-targeted data consistently closes perception gaps, it could accelerate VLM development cycles and reduce dependency on manually curated datasets. However, questions remain about whether synthetic training maintains robustness across diverse real-world conditions and whether this approach scales to more complex visual reasoning tasks beyond the 10 tested.

• →VisionFoundry generates synthetic VQA datasets from task keywords alone, improving VLM visual perception scores by 7-10% without human annotation.
• →The pipeline uses LLMs to create prompts and text-to-image models to synthesize training data, then validates consistency with a proprietary VLM.
• →VisionFoundry-10K dataset spans 10 visual perception tasks and shows favorable scaling properties as training data increases.
• →Results suggest limited task-targeted supervision is a key bottleneck in current VLM training, not fundamental model architecture limitations.
• →Synthetic supervision could reduce dependency on manually curated datasets and accelerate development of more robust multimodal models.