Internalizing Geometric Law: Learning from Solver Residuals for Precision-Critical Generation

Researchers introduce PyGeoX, a geometric constraint solver and benchmark that addresses hallucination problems in large language models for precision-critical tasks like technical design. They identify a failure mode called Outlier Gradient Masking in standard reward schemes and propose Saturating Additive Rewards (SAR) to improve constraint satisfaction, achieving 2.3x performance gains on hard problems.

This research tackles a fundamental problem in applying large language models to domains requiring rigorous constraint satisfaction. Precision-critical applications—technical diagramming, mechanical design, mathematical proofs—demand that outputs satisfy dozens of interconnected geometric constraints simultaneously, yet current LLMs frequently produce outputs violating these requirements. The team's core contribution lies not just in releasing open-source tools but in identifying why existing optimization approaches fail at scale.

The Outlier Gradient Masking phenomenon represents an important discovery about reward aggregation in constrained domains. When constraints are combined through global norms (such as exponential MSE), a single severe violation can dominate the gradient signal, preventing the model from learning partial progress on other constraints. This explains why geometry solvers plateau despite apparent optimization. SAR addresses this by decomposing rewards into bounded per-constraint terms, allowing models to maintain learning signals even when one constraint fails catastrophically.

The benchmark results demonstrate practical significance: an 8B parameter model using SAR matches performance of much larger frontier systems. This efficiency gain matters for deployment in resource-constrained environments and suggests that architectural innovation in reward design can outperform simple scaling. The open-source release of PyGeoX-Bench enables the community to develop and test new approaches systematically.

Future work will likely extend these insights to other constraint-heavy domains—theorem proving, circuit design, protein folding—where similar gradient masking phenomena probably occur. The methodology provides a template for making LLMs reliable in high-stakes technical applications where hallucination carries real costs.

• →Outlier Gradient Masking explains why standard reward schemes fail in multi-constraint geometric synthesis tasks
• →Saturating Additive Rewards improve hard-tier problem solving by 2.3x compared to MSE-based baselines
• →An 8B model with SAR achieves competitive performance with much larger frontier systems on geometric benchmarks
• →PyGeoX-Bench provides 300 stratified problems with verifiable per-constraint rewards for reproducible evaluation
• →The constraint decomposition approach generalizes beyond geometry to other precision-critical domains requiring strict verification