Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

Researchers present a signal-reshaping framework for GRPO (Group Relative Policy Optimization) that improves code-agent reinforcement learning under weak feedback conditions. The approach combines layered rewards, process-level credit assignment, and execution-aware rollout governance to increase strict compile-and-semantic accuracy from 38.5% to 53.5% on agentic code repair tasks.

This research addresses a fundamental challenge in training code-generation AI agents: feedback signals from code execution are often unreliable proxies for true task success. While compilation and execution provide objective binary signals, they don't capture whether code semantically solves the intended problem. The authors' contribution lies in decomposing GRPO's learning signal into three complementary components that work together to guide agent training more effectively.

The technical approach represents incremental but meaningful progress in reinforcement learning for code generation. By separating outcome rewards (which establish correct semantic ranking of solutions) from process signals (which assign credit within trajectories), the framework allows more fine-grained learning. The failure-cause-aware rollout governance ensures that comparisons between rollouts remain fair—a subtle but important detail often overlooked in RL systems. This decomposition is elegant because it doesn't require architectural changes to GRPO itself, making adoption straightforward.

The empirical results demonstrate concrete improvements across multiple metrics. The 13.9 percentage point jump in accuracy is substantial for code generation tasks, while the reduction in average evaluation steps from 23.50 to 17.02 indicates the agent learns more efficiently. The controlled ablations are particularly valuable, showing that each component contributes meaningfully rather than one dominating the improvements.

For the AI development community, this work validates that weak supervision can be effectively structured to improve agent training without requiring expensive human annotations or privileged information. Future code-agent systems will likely incorporate similar signal-reshaping techniques as standard practice, particularly as agents handle increasingly complex multi-step tasks in tool-use scenarios.

• →GRPO signal reshaping improves code-agent accuracy from 38.5% to 53.5% through layered rewards and process-level credit assignment
• →Outcome rewards and process signals must be reshaped separately to enable meaningful within-group comparisons in weak-feedback settings
• →Failure-cause-aware rollout governance ensures fairness when comparing solutions generated from identical prompts
• →Process-score weighting reduces evaluation steps by 27% while improving accuracy, indicating faster and more efficient agent learning
• →Token-level distillation alone is insufficient for long tool-use trajectories and cannot replace structured outcome and process signals