Residual Modeling for High-Fidelity Learned Compression of Scientific Data

Researchers present novel residual-centric compression methods (LBRC and NGLR) for scientific data that improve upon existing learned compression approaches by tailoring the encoding of reconstruction residuals to their structural properties. The techniques achieve 30-60% better compression ratios than Guaranteed Autoencoders and outperform the SZ compressor in high-fidelity regimes, addressing a critical bottleneck in compressing massive spatiotemporal datasets from scientific simulations.

This research addresses a fundamental challenge in scientific data management: compressing enormous spatiotemporal datasets from climate models, turbulence simulations, and weather reanalysis while maintaining accuracy. Learned compression methods have gained traction because they achieve higher compression ratios than traditional approaches, but they struggled when strict accuracy targets required per-block guarantees. Existing Guaranteed Autoencoder methods addressed this by retaining classical coefficients (SVD/PCA) until accuracy targets were met, creating a performance wall in the high-fidelity regime where the correction overhead dominated total bitrate.

The key insight driving this work is that learned residuals—the differences between original and reconstructed data—have fundamentally different statistical structures than the original scientific fields. Rather than forcing these residuals into traditional compression pipelines, the researchers designed representation schemes specifically for them. LBRC uses deterministic quantization and lossless encoding techniques including 3D Lorenzo differencing and entropy coding, requiring no neural network training. NGLR extends this with a causal neural predictor that improves entropy reduction while maintaining deterministic decoding properties critical for scientific reproducibility.

Results across three major scientific datasets (E3SM climate model, JHTDB turbulence database, ERA5 reanalysis) demonstrate substantial improvements: 30-60% better ratios over GAE and competitive or superior performance versus the widely-used SZ compressor. These gains matter significantly for climate research, materials science, and fluid dynamics communities processing petabyte-scale archives. The work enables learned compression to scale to production requirements where both compression efficiency and accuracy guarantees are non-negotiable. Future developments may extend these techniques to other structured residuals or explore hybrid approaches combining different predictor architectures.

• →Residual-centric compression achieves 30-60% better ratios than prior learned compression methods in high-fidelity regimes
• →NGLR adds neural prediction to deterministic pipelines, improving compression by 10-40% over LBRC while preserving reproducibility
• →Approach specifically targets scientific data with block-level NRMSE targets from 10^-6 to 10^-4, addressing prior scaling limitations
• →Methods eliminate the rate-dominance problem where correction streams consumed disproportionate bitrate in prior guaranteed compression approaches
• →Results validated across climate, turbulence, and weather datasets, demonstrating broad applicability to major scientific computing domains