Human-Guided Agentic AI for Multimodal Clinical Prediction: Lessons from the AgentDS Healthcare Benchmark

Researchers demonstrate that human-guided agentic AI systems outperform fully automated approaches on clinical prediction tasks, achieving strong benchmark results by combining domain expertise with autonomous workflows. The study reveals that human-directed decisions at critical junctures—particularly in multimodal feature engineering from clinical notes, billing documents, and vital signs—yield cumulative performance gains of +0.065 F1 over purely automated baselines.

This research addresses a critical gap in deploying AI systems for high-stakes healthcare applications where autonomous approaches often fall short of clinical requirements. The AgentDS Healthcare benchmark challenged teams to solve three distinct clinical prediction problems: hospital readmission, emergency department costs, and discharge readiness. The winning approach demonstrated that human expertise functions most effectively when strategically integrated into agentic workflows rather than either fully automating or manually controlling the entire process.

The healthcare sector has increasingly adopted AI tools, yet clinical predictions demand interpretability, reproducibility, and domain-specific validation that pure machine learning lacks. Previous approaches either relied on clinicians manually engineering features—labor-intensive and inconsistent—or delegated decisions entirely to automated systems, which frequently miss nuanced clinical patterns. This hybrid model represents a practical middle ground gaining traction across regulated industries.

The ablation studies provide actionable insights: multimodal feature extraction delivered the largest single improvement, suggesting that integrating diverse data sources (unstructured clinical text, scanned documents, time-series vitals) requires human judgment at extraction stages. Task-specific model selection and ensemble diversity, informed by clinical knowledge rather than random hyperparameter optimization, also drove measurable gains.

These findings have immediate implications for healthcare AI deployments. Organizations developing clinical decision support systems should prioritize human-in-the-loop architectures where domain experts guide feature engineering and validation rather than pursuing fully autonomous pipelines. The research validates a pattern emerging across high-consequence domains: agentic AI augments rather than replaces expert judgment, creating systems that remain interpretable and clinically defensible while leveraging computational efficiency.

• →Human-guided agentic AI outperformed fully automated approaches by +0.065 F1 on clinical prediction tasks through strategic intervention at key decision points.
• →Multimodal feature engineering from clinical notes, PDFs, and time-series data requires task-specific human judgment rather than one-size-fits-all extraction strategies.
• →Domain-informed ensemble diversity and deliberate model selection significantly outperformed automated hyperparameter search in clinical applications.
• →The hybrid human-AI approach achieved 5th place overall and 3rd place on discharge readiness, validating practical effectiveness in real healthcare benchmarks.
• →Interpretability, reproducibility, and clinical validity emerge as essential requirements driving adoption of guided agentic systems over fully autonomous alternatives.