From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging

Researchers propose Preference Delta Aggregation (PDA), a framework that combines weak preference signals from multiple smaller language model pairs into LoRA adapters, then merges them using Geometric Alignment Merging to improve larger models. The approach achieves 6.8-7.3 point improvements on knowledge reasoning and agentic search benchmarks by effectively composing complementary capabilities.

This research addresses a fundamental challenge in LLM training: the scarcity of high-quality supervised data. Rather than relying on single strong supervision signals, the team demonstrates that aggregating multiple weaker preference signals—derived from comparisons between modestly-sized model pairs—can produce stronger training outcomes when properly combined. The innovation lies in treating these signals as learnable deltas encoded in LoRA adapters, enabling compositional learning without retraining entire models.

The work builds on recent findings that relative quality comparisons from weak models contain meaningful supervisory information. However, naive aggregation of multiple preference signals risks directional interference, where conflicting gradients cancel benefits. The introduction of Geometric Alignment Merging solves this by aligning adapter subspaces before combination, ensuring diverse capability improvements complement rather than conflict with each other.

For the AI development community, this approach offers practical advantages: it reduces computational costs by leveraging smaller model pairs while improving larger models, and the modular adapter structure allows flexible composition without full model retraining. The 6.8-7.3 point improvements represent meaningful gains on complex reasoning tasks, suggesting the framework addresses real capability gaps. The empirical validation on knowledge reasoning and agentic search benchmarks indicates applicability across reasoning-intensive domains.

The methodology's broader significance lies in demonstrating that signal quality matters less than signal diversity when properly aggregated. Future applications likely include recursive application across model families, strategic selection of weak signal sources, and integration with existing alignment techniques to create more efficient training pipelines.

• →Preference Delta Aggregation enables improvement of larger models by composing multiple weak preference signals from smaller model pairs through LoRA merging.
• →Geometric Alignment Merging prevents directional interference by aligning adapter subspaces, enabling robust composition of diverse preference deltas.
• →The framework achieved 6.8-7.3 point average improvements on knowledge reasoning and agentic search benchmarks over single-delta baselines.
• →Multiple weak signals when properly aggregated outperform any individual preference signal, with gains increasing as more signals are incorporated.
• →The modular adapter-based approach reduces computational requirements while enabling flexible composition without full model retraining.