Before Kaggle, before the $1m Netflix prize, there was the KDD Cup, the first big annual machine learning competition. Starting in 1997, it brought together each year the world’s best predictive modelers. We now have over 25 years of experience of ML competitions to draw on, with tens of millions of dollars in prizes awarded. I will summarize what we’ve learned from this experience, and explain where, how, and why ML competitions can help advance machine learning research and practice.