SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation

Researchers introduce SC-ARENA, a new natural language evaluation framework for testing large language models in single-cell biology research. The framework addresses limitations in existing benchmarks by incorporating biological knowledge and real-world task formats to better assess AI models' understanding of cellular biology.

• →SC-ARENA creates a unified evaluation framework for testing LLMs on five core single-cell biology tasks including cell type annotation and perturbation prediction.
• →The framework introduces knowledge-augmented evaluation that incorporates external databases and scientific literature for biologically accurate assessments.
• →Current LLMs show uneven performance on complex biological tasks, particularly those requiring mechanistic or causal understanding.
• →The new evaluation method overcomes limitations of traditional string-matching metrics by providing interpretable, evidence-based judgments.
• →This work aims to guide development of more biology-aligned foundation models for scientific research applications.