Machine learning models often guide decisions that affect individuals, such as illness screening, mortality risk assessment, and treatment evaluation, often using data with noisy labels. We examine how label noise leads to unpredictable individual errors. We introduce the concept of regret, capturing errors from a single noise draw. Our results show that standard methods can perform well at the population level but still subject individuals to a lottery of mistakes. We propose techniques to identify these mistakes by training models on plausible noise draws. Our empirical study in clinical prediction highlights the difficulty of anticipating individual errors and suggests ways to improve safety by targeting these errors.