StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback

StableSketcher is a novel AI framework that enhances diffusion models for generating pixel-based hand-drawn sketches with improved prompt fidelity. The approach combines fine-tuned variational autoencoders with a reinforcement learning reward function based on visual question answering, alongside a new SketchDUO dataset of instance-level sketches paired with captions and Q&A pairs.

StableSketcher addresses a specific technical gap in generative AI: the difficulty of creating abstract, hand-drawn sketch representations through diffusion models. While diffusion models have revolutionized image generation broadly, they struggle with the sparse, minimalist characteristics of human sketches, which require different optimization strategies than photorealistic content. The researchers tackled this by optimizing the variational autoencoder's latent space specifically for sketch characteristics rather than using generic image optimization, a targeted approach that acknowledges domain-specific challenges in generative modeling.

The integration of visual question answering as a reward function for reinforcement learning represents a meaningful methodological advance. Rather than relying solely on traditional loss functions, this approach uses semantic reasoning about image content to ensure generated sketches maintain textual alignment and semantic consistency with prompts. This mirrors broader trends in AI development where multi-modal feedback mechanisms improve output quality.

The introduction of SketchDUO as the first instance-level sketch dataset paired with captions and question-answer pairs addresses a critical infrastructure gap. Existing sketch datasets typically use simple image-label pairs, limiting the training signals available for models. This richer annotation structure enables more nuanced training and evaluation. For developers working on sketch-based applications—design tools, architectural visualization, or educational platforms—StableSketcher's improvements in stylistic fidelity and prompt adherence could enable new product capabilities. The public release of code and dataset will accelerate downstream research and commercial applications in sketch generation.

• →StableSketcher improves diffusion model performance on abstract sketch generation through domain-specific VAE optimization and VQA-based reinforcement learning rewards.
• →The new SketchDUO dataset provides the first instance-level sketch annotations with captions and Q&A pairs, addressing limitations in existing sketch training data.
• →Visual question answering feedback mechanisms improve text-image alignment beyond traditional loss functions, suggesting broader applications in controlled generation tasks.
• →The framework demonstrates superior stylistic fidelity and prompt alignment compared to unmodified Stable Diffusion for sketch synthesis.
• →Public release of code and dataset will enable commercial applications in design tools, architectural visualization, and sketch-based workflows.