Optimal Bayesian Stopping for Efficient Inference of Consistent LLM Answers

Researchers propose a Bayesian stopping strategy that reduces LLM inference costs by up to 50% while maintaining answer accuracy. The method samples multiple LLM responses and stops once sufficient consistency is detected, using an efficient L-aggregated policy that tracks only the top 3 answer frequencies and achieves theoretical optimality.

This research addresses a fundamental efficiency problem in LLM deployment: the computational cost of sampling multiple responses to improve accuracy. While ensemble methods for LLMs have proven effective—particularly for math and reasoning tasks where consistency indicates correctness—they traditionally require fixed sampling budgets. The paper's innovation lies in applying Bayesian inference to determine when to halt sampling dynamically, eliminating wasteful additional queries once confidence thresholds are met.

The breakthrough centers on the L-aggregated stopping policy, which simplifies posterior computation by tracking only the L-1 most frequent answers rather than the full distribution. The theoretical finding that L=3 suffices for asymptotic optimality is notable: it demonstrates that a minimally informative aggregation strategy can match the performance of computationally expensive full Bayesian inference. This elegance makes the approach practically deployable across diverse LLM applications.

For organizations running inference at scale, this represents significant cost reduction without accuracy sacrifice. A 50% reduction in LLM API calls translates directly to operational expense reduction—critical for cost-sensitive applications like real-time reasoning systems or high-volume mathematical problem-solving. The method proves particularly valuable for API-based LLM services where per-query costs compound quickly.

The research opens pathways for further optimization in ensemble LLM strategies. Future work might explore adaptive sampling strategies that adjust consistency thresholds based on problem difficulty, or integration with other efficiency techniques like quantization or distillation. Implementation across production systems could establish new benchmarks for cost-efficient LLM deployment.

• →Bayesian stopping policy reduces LLM inference costs by up to 50% while preserving answer accuracy through dynamic sampling termination.
• →L-aggregated policy tracking only top 3 answer frequencies achieves asymptotic optimality with minimal computational overhead.
• →Method particularly effective for math and reasoning problems where answer consistency correlates with correctness.
• →Approach outperforms prior-free baselines while maintaining theoretical guarantees for optimal stopping behavior.
• →Practical optimization for large-scale LLM deployments and API-based services facing high per-query costs.