Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction

Researchers benchmark 12 LLMs under compression to evaluate whether quantization and pruning preserve uncertainty quantification alongside accuracy. The study reveals compression frequently decouples accuracy from uncertainty reliability, with smaller models absorbing compression-induced uncertainty poorly, suggesting current accuracy-only evaluation standards are insufficient for deployment readiness.

The research addresses a critical gap in LLM evaluation methodology by examining how model compression affects uncertainty calibration, not just accuracy metrics. While quantization and pruning reduce computational costs—essential for widespread deployment—the study demonstrates these techniques create unpredictable shifts in a model's ability to reliably assess its own confidence levels. This matters significantly because safety-critical applications like medical diagnosis, financial decision-making, or autonomous systems depend on calibrated uncertainty estimates as much as raw accuracy.

The findings reveal three important patterns: compression decouples accuracy preservation from uncertainty preservation, creating models that appear accurate but provide unreliable confidence signals; larger models demonstrate superior robustness to compression-induced uncertainty degradation compared to smaller variants; and uncertainty degradation follows threshold-like behavior rather than gradual decline, meaning compression effects can suddenly compound at specific points. This threshold behavior is particularly concerning for practitioners, as it suggests compression benefits cannot be smoothly traded against uncertainty costs.

For the AI development community, these results mandate methodological shifts in how compression pipelines are validated. Current industry practice evaluates compressed models primarily through accuracy benchmarks, potentially deploying unreliable uncertainty estimates into production systems. The implications extend to model selection and resource allocation decisions—practitioners cannot assume smaller, more compressed models simply trade accuracy for speed without accepting potentially severe uncertainty calibration losses. Organizations building safety-critical AI systems should adopt conformal prediction frameworks as standard validation components before deployment, fundamentally changing how model compression quality is assessed and benchmarked across the industry.

• →Model compression decouples accuracy from uncertainty quantification, meaning accurate compressed models may provide unreliable confidence estimates
• →Larger LLMs absorb compression-induced uncertainty far more effectively than smaller models, creating capability gaps between model sizes
• →Uncertainty degradation exhibits threshold-like behavior rather than gradual decline, making compression effects unpredictable and difficult to manage
• →Current accuracy-only evaluation standards are insufficient for assessing compressed LLM deployment readiness in safety-critical applications
• →Conformal prediction should become a standard benchmarking component in model compression pipelines across the industry