Joint Training Across Multiple Activation Sparsity Regimes

Researchers propose a novel neural network training strategy that cycles models through multiple activation sparsity regimes using global top-k constraints. Preliminary experiments on CIFAR-10 show this approach outperforms dense baseline training, suggesting joint training across sparse and dense activation patterns may improve generalization.

• →New training method cycles neural networks through different activation sparsity levels to improve generalization.
• →Approach uses global top-k constraints on hidden activations with progressive compression and periodic reset.
• →Initial results on CIFAR-10 with WRN-28-4 backbone show superior performance compared to dense training baselines.
• →Strategy is inspired by biological systems' stronger generalization capabilities across different activation regimes.
• →Two adaptive keep-ratio control strategies both demonstrated improved performance in single-run experiments.