Poster
in
Workshop: Mathematics of Modern Machine Learning (M3L)
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Anchit Jain · Rozhin Nobahari · Aristide Baratin · Stefano Sarao Mannelli
Keywords: [ fairness ] [ learning dynamics ] [ stochastic gradient descent ] [ online learning ] [ analytical model ] [ spurious correlation ]
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. This paper explores the evolution of bias in a teacher-student setup modeling different data sub-populations with a Gaussian-mixture model, by providing an analytical description of the stochastic gradient descent dynamics of a linear classifier in thissetting. Our analysis reveals how different properties of sub-populations influence bias at different timescales, showing a shifting preference of the classifier during training. We empirically validate our results in more complex scenarios by training deeper networks on real datasets including CIFAR10, MNIST, and CelebA.