4 articles tagged with #agent-benchmarks. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce Counsel, the first public meta-evaluation dataset for assessing how well LLM-based judges critique AI agent trajectories. The dataset addresses a critical bottleneck in agent evaluation by providing human-validated assessments of automated critique quality, enabling better calibration of evaluators at scale.
Researchers introduce MAVEN, a symbolic reasoning framework that improves language model generalization in tool-calling tasks by 23 percentage points (48% to 71% accuracy) on a new stress-test benchmark, while maintaining cost efficiency roughly 10x lower than frontier proprietary models. The work demonstrates that lightweight verification-centered scaffolds can enhance compositional reasoning without additional model training.
Researchers propose an outcome evidence reporting layer to improve the reliability of interactive agent benchmarks by explicitly tracking which runs have sufficient evidence of success versus uncertain cases. The framework evaluates five major AI benchmarks and reveals that surface-level outcome checks often fail to verify whether agents actually achieved intended results, making reported scores potentially misleading.
Researchers introduce Spatial Atlas, a compute-grounded reasoning system that combines deterministic spatial computation with large language models to create spatial-aware research agents. The framework demonstrates competitive performance on two benchmarks—FieldWorkArena for multimodal spatial question-answering and MLE-Bench for machine learning competitions—while improving interpretability by grounding reasoning in structured spatial scene graphs rather than relying on hallucinated outputs.
