Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR

Researchers propose Selective Eligibility Traces (S-trace), a new method for reinforcement learning that improves credit assignment in large language models by selectively identifying critical reasoning steps rather than uniformly crediting entire trajectories. The approach demonstrates performance gains of 0.49-3.16% across Qwen models while improving sample and token efficiency compared to existing critic-free algorithms.

The paper addresses a fundamental inefficiency in current reinforcement learning approaches for language models, specifically the uniform credit assignment problem in critic-free algorithms like GRPO. Traditional methods broadcast trajectory-level advantages equally across all tokens, failing to distinguish which reasoning steps actually contributed to correct outputs. This represents a real bottleneck in training efficiency for reasoning-focused LLMs.

The research builds on recent developments in verifiable reward learning, where models learn to solve complex reasoning tasks through reinforcement signals. By contextualizing GSPO within an eligibility traces framework, the authors identify it as a special case of uniform credit assignment. Their S-trace innovation implements sparse masking of low-entropy tokens to achieve fine-grained credit attribution, improving learning signal quality without requiring critic networks that add computational overhead.

The empirical results demonstrate meaningful improvements across model sizes, with particularly strong gains on mid-sized models (3.16% on 4B parameters) and sustained benefits at scale (2.98% on 8B). The simultaneous improvements in sample and token efficiency suggest the method achieves better data utilization, a critical metric for expensive LLM training.

This work affects AI developers building reasoning-capable models by offering a more sample-efficient training path. The method's compatibility with existing critic-free architectures enables straightforward adoption. Moving forward, the technique could influence how industrial training pipelines optimize language model reasoning capabilities, potentially reducing computational costs for achieving target performance benchmarks.

• →S-trace selectively credits important reasoning steps rather than uniformly crediting entire trajectories, improving learning efficiency
• →Empirical results show 0.49-3.16% performance gains across Qwen models with better sample and token efficiency
• →Method integrates with critic-free algorithms, avoiding additional computational overhead from critic networks
• →GSPO is theoretically framed as a special case of uniform credit assignment within the eligibility traces framework
• →Improvements persist across model scales from 1.7B to 8B parameters, suggesting broad applicability