Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regime

New research reveals that per-sample Adam optimizer's implicit bias differs significantly from full-batch Adam in machine learning training. The study shows incremental Adam can converge to different solutions than expected, potentially impacting AI model optimization strategies.

• →Per-sample Adam optimizer can deviate from full-batch Adam behavior, sometimes converging to ℓ2-max-margin instead of ℓ∞-max-margin classifiers
• →The implicit bias of Adam depends critically on both the batching scheme and the specific dataset being used
• →Researchers identified that incremental Adam's bias is characterized by a data-adaptive Mahalanobis-norm margin maximization
• →Signum optimizer maintains consistent ℓ∞-max-margin behavior regardless of batch size, unlike Adam
• →These findings challenge existing theoretical understanding of Adam optimizer in deep learning applications