L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling

Researchers document L20-Edu-135M, a 134.5M-parameter language model trained on a single NVIDIA L20 GPU using only 13 billion tokens—2.17% of the data used by comparable public models. While the model underperforms larger counterparts like SmolLM2, it achieves 87.1% of SmolLM-135M's performance with drastically reduced computational resources, offering insights into data-efficient small language model training.

This research presents a pragmatic case study in resource-constrained machine learning, addressing a growing gap between academic research capabilities and real-world deployment constraints. The L20-Edu-135M project demonstrates that meaningful language model performance is achievable with minimal computational overhead—a single consumer-grade GPU and 13 billion tokens—making it relevant for researchers and developers operating outside well-funded institutions.

The broader context reflects an industry-wide shift toward efficiency. As large language models dominate headlines, the practical demand for smaller, locally-deployable systems has intensified. This work contributes to the understudied space of optimal training recipes for resource-constrained regimes, providing architectural and data-curation details that practitioners can audit and replicate.

For developers and smaller organizations, the findings suggest that strategic data selection matters more than raw token volume. The model's use of cross-source deduplication, benchmark-overlap removal, and curated educational data demonstrates that thoughtful dataset engineering can partially compensate for reduced scale. However, the concerning result—that reinforcement learning from verifiable rewards (RLVR) degraded GSM8K performance from 1.82% to 1.21%—flags potential pitfalls in applying cutting-edge training techniques to resource-constrained settings.

The significance lies not in state-of-the-art performance claims but in transparency and reproducibility. By documenting the complete pipeline and releasing the checkpoint, researchers enable community validation and iteration. As edge AI and on-device inference gain importance, such auditable case studies establish baselines for what's achievable at different resource levels, informing future architecture and data strategy decisions.

• →L20-Edu-135M achieves 87.1% of SmolLM-135M performance using 2.17% of the training data through strategic curation and deduplication.
• →Single-GPU training with 13B tokens demonstrates feasible pathways for researchers with limited computational resources.
• →Reinforcement learning techniques degraded performance on math reasoning tasks, highlighting challenges in applying advanced training methods to constrained models.
• →Transparent documentation of architecture, data handling, and results enables community reproducibility and benchmarking.
• →Data quality and deduplication strategies appear more impactful than raw token volume in resource-constrained regimes.