Train Once, Answer All: Many Pretraining Experiments for the Cost of One

Researchers developed a method to conduct multiple AI training experiments simultaneously within a single pretraining run, reducing computational costs while maintaining research validity. The approach was validated across ten experiments using models up to 2.7B parameters trained on 210B tokens, with minimal impact on training dynamics.

• →Multiple pretraining experiments can now be conducted simultaneously during a single training run, significantly reducing computational costs.
• →The method was validated with ten experiments on models up to 2.7B parameters, replicating results from previous studies on data contamination and memorization.
• →Researchers successfully investigated knowledge acquisition, mathematical reasoning, and watermarking using this cost-efficient approach.
• →The approach had minimal influence on model training dynamics and overall performance across experiments.
• →A new technique called continual pretraining dependence testing (CPDT) was developed to detect negligible interactions between experiments.