Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training

Researchers demonstrate that Forward-Forward (FF) layer-local learning, a biologically-plausible alternative to backpropagation, significantly underperforms on real-world image datasets despite closing gaps on synthetic benchmarks. The study reveals a critical scaling limitation: FF reaches only 49.4% accuracy at ImageNet-100 224x224 resolution versus 75%+ for standard backpropagation, undermining claims that layer-local training represents a viable alternative for realistic deep learning applications.

The paper challenges recent optimism around Forward-Forward algorithms by conducting rigorous real-data evaluation that exposes fundamental limitations invisible in prior synthetic experiments. While FF-based methods have attracted attention as biologically plausible alternatives to backpropagation—potentially offering benefits like reduced memory footprint and local learning rules—this research provides empirical evidence that these theoretical advantages don't translate to practical improvements at meaningful scales.

The researchers developed DTG-FF, incorporating dynamic temperature goodness and multi-layer fusion optimizations, achieving state-of-the-art FF performance across nine benchmarks. However, performance degradation becomes severe as problem complexity increases: the FF-BP gap widens from 2.40 percentage points on CIFAR-10 to 5.93 points on CIFAR-100, and reaches a dramatic 25.6-point deficit at 224x224 ImageNet resolution. This pattern suggests FF's core mechanism struggles with fine-grained discrimination tasks that characterize realistic computer vision problems.

The systems audit further undermines FF's practical justification. Rather than enabling memory savings on commodity hardware, DTG-FF actually consumed 7.90 GB versus BP's 4.18 GB while processing fewer images per second (138 vs. 157 imgs/s). This contradicts arguments that FF offers computational advantages under fair comparison conditions.

The synthetic-versus-real discrepancy is particularly important: FF outperforms BP on synthetic teacher-student tasks as class count increases, yet reverses on real images. This reveals that synthetic K-sweeps confound output dimensionality with actual discrimination difficulty, systematically overstating FF's transferability. The finding suggests recent FF enthusiasm may rest on misleading benchmarks rather than fundamental algorithmic advantages.

• →Forward-Forward layer-local learning shows a 25.6 percentage point accuracy gap versus backpropagation at ImageNet-224×224 scale (49.4% vs. 75%), revealing a real-data ceiling invisible in 32×32 benchmarks
• →Synthetic benchmarks systematically overstate FF performance by confounding output dimensionality with actual discrimination difficulty, causing K-conflict results to reverse sign on real images
• →Memory-based justifications for FF are unsupported on commodity 8GB hardware, where DTG-FF consumes 1.9× more memory and processes 12% fewer images per second than standard backpropagation
• →FF performance degradation accelerates with task complexity, widening from 2.40pp on CIFAR-10 to 5.93pp on CIFAR-100, indicating fundamental limitations rather than implementation issues
• →The gap between synthetic-task performance and real-image results suggests recent FF enthusiasm may reflect misleading benchmarks rather than viable algorithmic alternatives to backpropagation