#pass-at-k News & Analysis

3 articles tagged with #pass-at-k. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBullisharXiv – CS AI · Jun 97/10

🧠

More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)

Researchers propose Reset-and-Discard (ReD), a novel querying method that improves large language model inference efficiency by optimizing the coverage@cost metric—the number of unique questions answered within a fixed budget. The technique reduces computational attempts, tokens, and financial costs needed to achieve desired performance levels across coding, math, and reasoning tasks.

AINeutralarXiv – CS AI · Jun 196/10

🧠

Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

Researchers identify a critical blind spot in pass@k, the standard metric for evaluating math reasoning difficulty in large language models. Their analysis reveals that 10-23% of problems marked as unsolvable through sampling can actually be solved using deterministic inference with activation grafting perturbations, suggesting current difficulty assessments systematically underestimate model capabilities.

AIBullisharXiv – CS AI · Jun 116/10

🧠

Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

Researchers introduce Pass@K Policy Optimization (PKPO), a reinforcement learning method that optimizes for multiple solution attempts jointly rather than individually, enabling better exploration and problem-solving on harder tasks. The approach derives unbiased estimators for pass@k performance across arbitrary k values and demonstrates improved learning on challenging benchmarks using open-source LLMs.