Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage

Researchers introduce BioConCal, a supervised scoring system that evaluates biomedical entity candidates surfaced by multiple LLMs across five public datasets. The tool improves candidate verification from 75.3% to 91% AUROC by leveraging agreement patterns and document features, enabling more efficient curator review workflows rather than recovering missed entities.

The biomedical NLP research community faces a persistent challenge: while large language models excel at surfacing plausible biomedical entity mentions, distinguishing corpus-convention correctness from mere surface-level plausibility remains computationally expensive. This paper addresses that gap by reframing entity validation as a candidate-triage problem rather than a standalone extraction task.

The work emerges from recognition that multi-LLM agreement, though intuitively appealing as a confidence signal, doesn't reliably indicate annotation-standard correctness. Biomedical NER involves navigating complex terrain—entity span boundaries, granularity levels, and domain-specific type schemas vary across annotation conventions. The authors built BioConCal as an in-domain supervised scorer operating on a master candidate table created by aligning eight LLMs' outputs across five established datasets. Rather than seeking missing entities, BioConCal reshapes noisy panel output into a higher-yield review queue.

The performance metrics illustrate practical value: at a validation-selected 0.95 precision threshold, BioConCal selects 1,340 candidates with empirical 93.9% precision, versus only 293 for raw agreement scoring. This 4.5x increase in candidate volume while maintaining precision targets significantly reduces curator workload. The approach acknowledges its limitations—entity-type distribution shifts require target-domain validation, and final character localization remains a separate deterministic step.

For biomedical AI development, this methodology signals a maturing field moving beyond raw extraction metrics toward practical curation workflows. Organizations building biomedical knowledge bases can leverage panel-based scoring to optimize human-in-the-loop annotation pipelines, reducing both computational overhead and annotation costs.

• →BioConCal improves AUROC from 75.3% to 91% for biomedical entity candidate verification using multi-LLM agreement patterns and surface features
• →Multi-LLM agreement alone is insufficient for corpus-convention correctness; supervised scoring better captures annotation standard compliance
• →At target precision thresholds, the system increases candidate volume 4.5x compared to raw agreement scoring, enhancing curator efficiency
• →The approach reshapes noisy panel streams into higher-yield review queues rather than primarily recovering universally-missed entities
• →Entity-type distribution shifts require target-domain validation, limiting direct cross-domain transfer of trained models