CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents

CodeClinic introduces a benchmark for evaluating whether large language model agents can autonomously generate clinical skills rather than relying on pre-built tool libraries. The research demonstrates that an offline autoformalization pipeline converting clinical guidelines into Python libraries improves consistency and reduces token usage by 40% compared to zero-shot code generation.

CodeClinic addresses a fundamental limitation in clinical AI systems: the dependency on manually maintained tool libraries that require constant expert curation. As healthcare institutions increasingly adopt LLM-based monitoring systems for tasks like ICU surveillance and patient tracking, the scalability bottleneck becomes acute. This research tackles that problem by enabling agents to synthesize their own clinical reasoning tools from natural language guidelines, eliminating the need for handcrafted skill repositories that become outdated and institution-specific.

The benchmark itself represents a significant methodological contribution, combining two complementary evaluation paradigms. The longitudinal ICU surveillance task mirrors real-world clinical workflows with structured decision points across 25 findings, while the compositional information seeking task spans 63,000 instances to stress-test multi-step reasoning chains. This stratified approach—measuring performance against compositional dependency depth—reveals how well agents handle increasing complexity, a critical measure for clinical reliability.

The autoformalization pipeline's 40% token reduction has immediate practical implications. In clinical settings operating at scale, token efficiency translates directly to cost reduction and lower latency for time-sensitive monitoring tasks. The verification aspect—ensuring converted skills maintain clinical validity—addresses a critical safety concern that has plagued automated clinical reasoning systems.

The work signals a shift from static tool-building toward dynamic skill synthesis in healthcare AI. Institutions could eventually maintain clinical guidelines as source material rather than curated code libraries, reducing friction between policy updates and system implementation. Future research should examine how well these techniques generalize across institutions with different clinical protocols and whether the approach scales to more complex multi-organ patient scenarios.

• →CodeClinic enables LLM agents to autonomously generate reusable clinical skills from guidelines instead of relying on manually curated toolboxes.
• →The autoformalization pipeline improves token efficiency by 40% while maintaining consistency compared to zero-shot code generation.
• →Benchmark evaluation spans 63,000 instances across nine domains with stratified complexity levels for comprehensive agent assessment.
• →The approach addresses healthcare's scalability challenge by reducing expert effort required to maintain institution-specific clinical tool libraries.
• →Verification mechanisms within the pipeline ensure converted clinical guidelines maintain medical validity and safety standards.