Do Machines Fail Like Humans? A Human-Centred Out-of-Distribution Spectrum for Mapping Error Alignment

Researchers propose a human-centered framework for evaluating whether AI systems fail in ways similar to humans by measuring out-of-distribution performance across a spectrum of perceptual difficulty rather than arbitrary distortion levels. Testing this approach on vision models reveals that vision-language models show the most consistent human alignment, while CNNs and ViTs demonstrate regime-dependent performance differences depending on task difficulty.

This research addresses a fundamental challenge in trustworthy AI: determining whether model-human parity in accuracy reflects genuine cognitive alignment or merely superficial performance matching. The conventional approach of testing AI systems on distorted or out-of-distribution data has lacked principled methodology, using either model-training-relative definitions or arbitrary distortion parameters disconnected from human perception. The proposed framework reorients the problem around human perceptual difficulty, creating a measurable spectrum that enables direct comparison across comparable challenge levels.

The work builds on growing recognition that AI trustworthiness requires understanding decision-making processes, not just output accuracy. Prior research established that models and humans often fail differently on identical tasks, but lacked structured methods to quantify these differences across varying difficulty conditions. By anchoring out-of-distribution definitions to human accuracy rather than arbitrary metrics, this research provides a more rigorous comparative foundation.

For AI developers and researchers, these findings carry practical implications. The observation that different architectures show regime-dependent alignment—CNNs excelling in near-OOD scenarios while ViTs perform better in far-OOD conditions—suggests no universal architecture achieves human-like robustness across all difficulty levels. Vision-language models' consistent alignment across conditions points toward multimodal approaches as promising for human-centered AI development.

Future work should examine whether this framework generalizes beyond vision to language and multimodal domains, and whether understanding alignment patterns can guide architectural improvements. The research also raises questions about whether alignment at human difficulty levels is actually desirable for safety-critical applications, suggesting alignment alone may not be sufficient for trustworthy deployment.

• →Human-centered OOD framework redefines distortion difficulty by human perceptual ability rather than arbitrary parameters
• →Vision-language models demonstrate most consistent human alignment across near and far out-of-distribution conditions
• →CNN and ViT performance profiles differ by difficulty regime, with CNNs favoring near-OOD and ViTs favoring far-OOD
• →Model-human error alignment varies significantly by difficulty level, requiring calibrated comparisons for principled assessment
• →Framework enables more rigorous evaluation of whether AI systems replicate human decision-making processes