"Skill issues'': data-centric optimization of lakehouse agents

Researchers present a data-centric optimization framework for AI coding agents operating on branching lakehouses, demonstrating that agent skills can be systematically improved through task-verifier pairs and sandboxed execution. The approach treats agent evaluation as state verification rather than output matching, achieving 31.9% accuracy improvements on preliminary tasks.

This research addresses a fundamental challenge in AI agent development: optimizing the auxiliary artifacts—skills and environment files—that determine agent performance beyond base model quality. Rather than focusing solely on improving the underlying language model, the authors recognize that agents require specialized knowledge about how to interact with specific data infrastructure systems. The branching lakehouse architecture, exemplified by Bauplan, provides an ideal testbed because it exposes data operations through code-like primitives (branches, commits, merges), making agent actions inspectable and verifiable.

The shift from output-matching to state-verification represents a meaningful methodological advance. Traditional agent evaluation often relies on comparing final outputs to expected results, but this approach struggles with data workflows where intermediate states matter significantly. By examining concrete lakehouse changes induced by agent-generated code, researchers can assess whether agents correctly manipulated data structures, regardless of whether the final output format matches expectations perfectly.

The 31.9% accuracy improvement on 25 tasks suggests that agent performance remains significantly constrained by environment knowledge rather than model capabilities alone. This finding has implications for practitioners building production AI agents: optimizing task-verifier pairs and providing comprehensive skills documentation may deliver better returns than pursuing increasingly larger foundation models.

The research points toward a broader trend where infrastructure-aware optimization becomes critical for deploying agents in specialized domains. As coding agents integrate deeper with enterprise data systems, the ability to systematically improve agent-infrastructure fit through sandboxed testing and programmatic verification becomes a competitive advantage.

• →Agent performance depends critically on skills and environment files, not just model quality
• →Branching lakehouse architecture enables state-verification evaluation of agent-generated code
• →Data-centric optimization pipeline achieved 31.9% accuracy improvement across 25 preliminary tasks
• →Write-path data workflows provide better optimization substrates than read-only task evaluation
• →Sandboxed execution and programmatic lakehouse state checks enable systematic agent skill improvement