AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models

Researchers introduce AlphaForgeBench, a new evaluation framework that addresses critical instability issues in Large Language Models deployed as trading agents. Rather than having LLMs generate discrete trading actions, the framework redefines their role as quantitative researchers producing alpha factors and strategies, enabling deterministic, reproducible evaluation aligned with real-world financial workflows.

Current LLM-based trading benchmarks suffer from fundamental reliability problems that undermine their validity as evaluation tools. Large Language Models exhibit severe run-to-run variance, generate inconsistent action sequences even under deterministic decoding, and frequently flip decisions between adjacent time steps—behaviors rooted in their stateless autoregressive architecture and sensitivity to continuous-to-discrete action mapping in portfolio contexts. These instabilities make existing online and offline trading simulations unreliable for assessing LLM capability in financial decision-making.

The proliferation of financial benchmarks reflects genuine interest in evaluating LLMs for quantitative finance, but most frameworks implicitly treat language models as stochastic trading agents rather than reasoning systems. This mismatch between capability and task design creates fundamental evaluation problems that no amount of additional testing can overcome. The stateless nature of autoregressive models means each token prediction lacks persistent memory of prior actions, making sequential portfolio decisions inherently unstable.

AlphaForgeBench reframes the problem by repositioning LLMs where they demonstrate actual strength: financial reasoning and hypothesis formulation. By requiring models to generate executable alpha factors and compose factor-based strategies instead of producing discrete trading signals, the framework decouples reasoning from execution mechanics. This approach yields deterministic, reproducible results while remaining grounded in authentic quantitative research workflows where domain experts generate factors that execution systems operationalize.

For the AI-finance research community, this work establishes methodological rigor around LLM evaluation in trading contexts. The framework's success across multiple state-of-the-art models suggests alpha factor generation represents a more appropriate benchmark task than real-time action generation, potentially redirecting research toward genuinely useful applications rather than demonstrating unstable agent behavior.

• →LLMs exhibit severe behavioral instability in sequential trading decisions due to their stateless autoregressive architecture and sensitivity to action mapping in portfolio tasks.
• →AlphaForgeBench repositions LLMs as quantitative researchers generating alpha factors rather than stochastic trading agents making discrete actions.
• →The framework achieves deterministic, reproducible evaluation by decoupling reasoning from execution mechanics.
• →Existing trading benchmarks fundamentally underestimate execution-induced instability and therefore provide unreliable capability assessments.
• →Factor-based strategy generation aligns better with real-world quantitative research workflows and LLM strengths than direct action generation.