On the Equivalence of Random Network Distillation, Deep Ensembles, and Bayesian Inference

Researchers establish theoretical connections between Random Network Distillation (RND), deep ensembles, and Bayesian inference for uncertainty quantification in deep learning models. The study proves that RND's uncertainty signals are equivalent to deep ensemble predictive variance and can mirror Bayesian posterior distributions, providing a unified theoretical framework for efficient uncertainty quantification methods.

• →Random Network Distillation's squared self-predictive error is mathematically equivalent to the predictive variance of deep ensembles in infinite network width limits.
• →RND error distributions can be constructed to mirror Bayesian posterior predictive distributions through specific target function design.
• →The research introduces a posterior sampling algorithm that generates exact Bayesian posterior samples using modified Bayesian RND models.
• →The findings unify three major uncertainty quantification approaches under a single theoretical framework using neural tangent kernel analysis.
• →This theoretical foundation opens new pathways for computationally efficient yet rigorous uncertainty quantification in deep learning deployments.