Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

Researchers demonstrate that low-bit quantization of reasoning models introduces a hidden cost: quantized models generate significantly longer chains of thought to maintain accuracy, offsetting per-token speedup gains. The study introduces metrics to measure this token inflation and finds quantization-aware training as the most effective mitigation strategy.

Quantization has become a standard technique for reducing LLM inference costs, but this research reveals a critical blind spot in how the AI community evaluates quantized reasoning models. While INT4/INT3 quantization preserves final-answer accuracy, the models compensate by generating substantially longer reasoning traces—effectively trading one cost for another. This finding challenges the assumption that accuracy metrics alone sufficiently capture quantization effects on complex reasoning tasks.

The research emerges from the broader push to democratize access to advanced reasoning models like o1 and similar systems. As these models become computationally expensive, practitioners eagerly deploy quantization to reduce serving costs. However, this study's introduction of the CoT Token Inflation Ratio reveals that real-world deployment costs may not improve as expected when token volume increases alongside per-token speedup.

For organizations deploying quantized reasoning models in production, this creates a crucial economics problem. A model that maintains accuracy while doubling token generation actually increases total inference cost, latency, and user-facing delays—defeating the primary justification for quantization. This particularly impacts applications like agentic tool-use systems where chain-of-thought length directly affects end-to-end latency and API costs.

The findings point toward quantization-aware training as the path forward, suggesting the industry must shift from post-hoc quantization to more sophisticated approaches that preserve reasoning efficiency alongside accuracy. Organizations should demand token-length reporting alongside accuracy metrics when evaluating quantized models, establishing new benchmarking standards that reflect real-world serving costs rather than theoretical per-token improvements.

• →Low-bit quantization preserves accuracy but increases reasoning-token generation, creating a hidden cost that offsets per-token speedup benefits.
• →Token inflation is accompanied by behavioral changes including more intermediate steps and semantic repetition in quantized model reasoning traces.
• →Quantization-aware training shows more promise than prompting or decoding-time sampling for mitigating both accuracy degradation and token inflation.
• →Real-world serving penalties from token inflation are measurable and significant enough to outweigh quantization benefits in production deployments.
• →The AI community should report token-length metrics alongside accuracy when evaluating quantized reasoning models to capture true inference costs.