Poster Session
in
Workshop: Scientific Methods for Understanding Neural Networks
How Learning Rates Shape Neural Network Focus: Insights from Example Ranking
Ekaterina Lobacheva · Keller Jordan · Aristide Baratin · Nicolas Le Roux
Abstract:
The learning rate is a key hyperparameter that affects both the speed of training and the generalization performance of neural networks. Through a new loss-based example ranking analysis, we show that networks trained with different learning rates focus their capacity on different parts of the data distribution, leading to solutions with different generalization properties. These findings, which hold across architectures and datasets, provide new insights of how learning rates affect model performance and example-level dynamics in neural networks.
Chat is not available.