Eliciting Numerical Predictive Distributions of LLMs Without Autoregression

Researchers developed a method to extract numerical prediction distributions from Large Language Models without costly autoregressive sampling by training probes on internal representations. The approach can predict statistical functionals like mean and quantiles directly from LLM embeddings, potentially offering a more efficient alternative for uncertainty-aware numerical predictions.

• →LLMs can be applied to regression tasks but autoregressive decoding is computationally expensive for numerical outputs
• →Regression probes can predict statistical functionals of LLM output distributions directly from internal representations
• →LLM embeddings contain informative signals about summary statistics and numerical uncertainty
• →This method could provide lightweight alternatives to sampling-based approaches for numerical predictions
• →The research opens questions about how LLMs internally encode uncertainty in numerical tasks