#humaneval News & Analysis

5 articles tagged with #humaneval. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

5 articles

AIBullisharXiv – CS AI · Jun 97/10

🧠

MAR:Multi-Agent Reflexion Improves Reasoning Abilities in LLMs

Researchers present Multi-Agent Reflexion (MAR), a technique that improves LLM reasoning by using multiple AI agents with distinct personas to debate and generate diverse reflections rather than having a single model reflect on itself. The approach achieves 47% accuracy on HotPotQA and 82.7% on HumanEval, outperforming traditional single-agent reflection methods that suffer from repetitive error patterns.

AIBearisharXiv – CS AI · Apr 137/10

🧠

On the Limits of Layer Pruning for Generative Reasoning in Large Language Models

Research demonstrates that layer pruning—a compression technique for large language models—effectively reduces model size while maintaining classification performance, but critically fails to preserve generative reasoning capabilities like arithmetic and code generation. Even with extensive post-training on 400B tokens, models cannot recover lost reasoning abilities, revealing fundamental limitations in current compression approaches.

AINeutralarXiv – CS AI · Jun 26/10

🧠

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

A paired study comparing six multi-agent LLM architectures across 1,968 code generation tasks reveals that architectural complexity increases code structural complexity by 50-130% without improving functional accuracy. The research demonstrates that simpler orchestration pipelines match or exceed performance of elaborate multi-agent systems, challenging assumptions about architectural elaboration in AI code generation.

🧠 GPT-4

AINeutralarXiv – CS AI · May 126/10

🧠

Sketch-and-Verify: Structured Inference-Time Scaling via Program Sketching

Sketch-and-Verify is an inference-time scaling technique that improves small language model performance by having the LLM generate multiple algorithmic strategies as program sketches, then filling and verifying them. On HumanEval+, this approach delivers superior cost-performance within a model tier compared to flat sampling, though upgrading to a stronger model tier remains more effective than scaling test-time compute on smaller models.

🧠 Gemini

AINeutralHugging Face Blog · Jun 184/104

🧠

BigCodeBench: The Next Generation of HumanEval

The article appears to discuss BigCodeBench as a new evaluation benchmark for code generation, positioning it as an advancement over HumanEval. However, the article body is empty, preventing detailed analysis of its features, methodology, or potential impact on AI development.