Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems

A new arXiv paper audits 30 LLM-based trading studies and finds that while agent architectures are well-documented, evaluation methodologies—including execution timing, transaction costs, and data splits—lack standardization, making performance claims difficult to compare or reproduce. The authors argue that LLM trading research needs clearer reporting standards for execution realism before architectural improvements can be meaningfully assessed.

The proliferation of LLM-based trading systems has outpaced the development of rigorous evaluation frameworks, creating a credibility gap in the research community. This arXiv paper addresses a critical blind spot: most published studies focus heavily on model architecture while glossing over the execution assumptions that determine whether backtested results translate to real-world profitability. The audit reveals inconsistencies in point-in-time data controls, temporal split discipline, turnover treatment, and transaction-cost modeling across the sampled literature.

This reproducibility crisis mirrors challenges seen in machine learning more broadly, where architectural novelty often eclipses methodological rigor. In financial trading, the stakes are higher because small assumptions about slippage, market impact, and execution timing can compress reported returns by significant margins. The authors' 10-equity worked example demonstrates this concretely: explicit friction modeling materially changes strategy performance.

For the AI-crypto trading ecosystem, this paper signals growing maturity in critical evaluation. Venture capital and institutional investors increasingly scrutinize LLM trading claims, and this work provides a framework for separating credible research from overstated results. Developers building trading agents now have a checklist of reporting standards to adopt, while researchers can benchmark their own disclosure practices against the audit matrix.

The path forward requires community adoption of standardized reporting templates covering execution semantics, universe definition, cost assumptions, and artifact release. Without this baseline, LLM trading research risks repeating the hype-and-disappointment cycle common to algorithmic trading. The next wave of credibility will come not from better models, but from transparent, reproducible evaluation.

• →LLM trading research excels at documenting agent architecture but fails to standardize execution assumptions, making performance claims difficult to verify or compare.
• →Transaction costs, execution timing, and turnover treatment can materially compress reported returns, yet these details are often underspecified in published studies.
• →A reproducibility audit of 30 studies found inconsistent disclosure across point-in-time controls, data splits, and cost modeling.
• →Clearer reporting standards for execution realism and evaluation transparency are now more urgent than incremental improvements to agent design.
• →Institutional investors and developers need a standardized checklist to evaluate LLM trading claims against realistic market conditions.