Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

A large-scale study of 62,808 AI safety evaluations across six frontier models reveals that deployment scaffolding architectures can significantly impact measured safety, with map-reduce scaffolding degrading safety performance. The research found that evaluation format (multiple-choice vs open-ended) affects safety scores more than scaffold architecture itself, and safety rankings vary dramatically across different models and configurations.

• →Map-reduce scaffolding degrades measured AI safety with a number needed to harm of 14, while other scaffold architectures preserve safety within acceptable margins.
• →Switching from multiple-choice to open-ended evaluation formats shifts safety scores by 5-20 percentage points, exceeding any scaffold effects.
• →Model-scaffold interactions vary dramatically, with safety performance changes ranging from -16.8 to +18.8 percentage points on the same benchmark.
• →AI safety rankings reverse completely across different benchmarks, making universal safety claims unreliable.
• →The study establishes that per-model, per-configuration testing is necessary as no composite safety index achieves reliable generalizability.