Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting

Researchers propose the Beta-Bernoulli Calibrator (BBC), a novel method that improves large language model forecasting by converting point estimates into probability distributions using both binary outcomes and aggregated human forecast signals. The approach demonstrates better calibration and accuracy than existing post-hoc methods while leveraging epistemic uncertainty as a more reliable error predictor than verbalized confidence.

The Beta-Bernoulli Calibrator addresses a fundamental limitation in current LLM forecasting systems: they typically learn from binary outcomes alone, ignoring the rich information embedded in human crowd forecasts. This oversight represents a missed opportunity, as aggregated human predictions contain both probability estimates and metadata about forecaster agreement that signal underlying uncertainty. By modeling event likelihood as a Beta distribution and outcomes as Bernoulli variables, BBC captures epistemic uncertainty through variance—offering more nuanced probability estimates than traditional confidence statements. The research demonstrates that this approach outperforms both classical calibration methods and models fine-tuned specifically for forecasting tasks, suggesting a fundamental advantage to the probabilistic framework. Importantly, BBC remains computationally lightweight and generalizes well across different scenarios, reducing implementation barriers for adoption. The finding that epistemic uncertainty better predicts forecasting error than verbalized confidence has significant implications for AI reliability assessment. Rather than relying on LLMs to articulate confidence levels—a notoriously problematic approach—this method derives uncertainty directly from probability distributions fitted to empirical data. This shift from qualitative confidence statements to quantitative uncertainty measures represents a meaningful advancement in AI trustworthiness. The work bridges machine learning and human collective intelligence, leveraging forecast aggregation insights from prediction market research and applying them to LLM calibration. As organizations increasingly deploy LLMs for consequential decisions, robust uncertainty quantification becomes critical infrastructure. BBC's demonstrated generalization across diverse forecasting tasks suggests practical applicability beyond academic benchmarks.

• →Beta-Bernoulli Calibrator converts LLM point forecasts into calibrated probability distributions using both outcomes and human forecast signals.
• →BBC captures epistemic uncertainty through variance, providing more reliable error prediction than LLM-generated confidence statements.
• →The method outperforms traditional post-hoc calibration and task-specific fine-tuning while remaining lightweight and generalizable.
• →Aggregated human forecasts contain underutilized information about agreement and uncertainty that improves model calibration.
• →Uncertainty quantification through probabilistic modeling addresses a critical need in deploying LLMs for high-stakes forecasting applications.