Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark

Researchers benchmark supervised fine-tuned vision-language models against frontier zero-shot AI baselines on screen-conditioned action prediction using the PiSAR dataset. A fine-tuned Qwen3-VL-8B model substantially outperforms GPT and Claude zero-shot approaches (0.783 vs 0.459-0.482 semantic similarity), but the same training recipe fails on Gemma-4-26B, revealing critical architecture-to-method misalignment in model optimization.

This technical benchmark exposes a fundamental challenge in AI model development: frontier capabilities do not guarantee fine-tuning responsiveness. The PiSAR study demonstrates that smaller, instruction-tuned models can dramatically exceed larger reasoning-optimized systems when aligned with task-specific training recipes. The Qwen3-VL-8B's 0.30 absolute improvement over GPT-5.5 and Claude Opus, coupled with 79% performance at semantic similarity thresholds versus 1-2% for zero-shot baselines, suggests that architectural design choices and pretraining objectives significantly constrain downstream adaptation potential.

The research contextualizes a growing recognition in machine learning that parameter count and frontier benchmarks do not predict fine-tuning efficacy. Reasoning-tuned models like Gemma-4-26B appear to develop brittle feature hierarchies resistant to task-specific adjustment, while instruction-tuned multimodal models retain greater plasticity. This finding challenges scaling assumptions that larger models universally benefit from supervised adaptation.

For practitioners deploying vision-language systems in production environments—particularly screen-understanding tasks in UI automation, accessibility, and e-commerce—architecture selection now ranks alongside data quality. Organizations pursuing custom action-prediction systems face architectural constraints that cannot be overcome through training data volume alone. The failure of Gemma-4-26B to scale improvements despite identical training conditions indicates diminishing returns on parameter expansion without concurrent architectural innovation.

Future work should investigate transfer learning mechanisms across model families and determine whether stronger fine-tuning methods can overcome inherent model resistance. The mismatch suggests that architecture-aware training strategies, not universal recipes, drive performance gains in specialized domains.

• →Fine-tuned Qwen3-VL-8B outperforms frontier zero-shot baselines by 0.30 semantic similarity on screen-conditioned action prediction tasks
• →Identical training recipes produce divergent results across architectures, with Gemma-4-26B underperforming Qwen despite larger parameter count
• →Reasoning-tuned models show resistance to task-specific fine-tuning, suggesting architectural design constrains downstream optimization
• →Model selection for specialized domains requires architecture-aware evaluation rather than reliance on frontier capability benchmarks
• →The PiSAR benchmark reveals that instruction-tuned multimodal systems retain greater plasticity than parameter-optimized reasoning models