Agency and Architectural Limits: Why Optimization-Based Systems Cannot Be Norm-Responsive

New research demonstrates that AI systems trained via RLHF cannot be governed by norms due to fundamental architectural limitations in optimization-based systems. The paper argues that genuine agency requires incommensurable constraints and apophatic responsiveness, which optimization systems inherently cannot provide, making documented AI failures structural rather than correctable bugs.

• →Optimization-based AI systems like RLHF-trained LLMs are formally incompatible with normative governance due to their scalar metric approach.
• →Genuine agency requires two conditions that optimization systems cannot satisfy: incommensurable constraints and apophatic responsiveness.
• →Common AI failure modes like sycophancy and hallucination are structural manifestations, not correctable training bugs.
• →The Convergence Crisis occurs when humans verifying AI outputs under pressure become optimizers themselves, eliminating normative accountability.
• →The research provides architectural specifications for what qualifies as an agent versus a sophisticated instrument.