TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models

Researchers have introduced the TACIT Benchmark, a new programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains for evaluating AI models. The benchmark offers both generative and discriminative evaluation tracks with 6,000 puzzles and 108,000 images, using deterministic verification rather than subjective scoring methods.

• →TACIT Benchmark introduces a new standardized way to evaluate AI visual reasoning capabilities across 6 domains including spatial navigation and logical constraint satisfaction.
• →The benchmark provides dual evaluation tracks - generative solutions verified by computer vision and multiple choice with carefully designed distractors.
• →Version 0.1.0 includes 6,000 puzzles with 108,000 PNG images across three resolutions, all available under Apache 2.0 license.
• →Each distractor violates exactly one structural constraint, requiring models to reason about fine-grained visual differences.
• →The benchmark addresses limitations of existing visual reasoning tests that rely on subjective LLM-as-judge scoring procedures.