Machine learning has begun to revolutionize almost all areas of health research. Success stories cover a wide variety of application fields ranging from radiology and dermatology to gastroenterology and mental health applications. Strikingly, however, such widely known success stories appear to be lacking in some subfields of healthcare, such as surgery. A main reason for this phenomenon could be the lack of large annotated training data sets. In the past years, we have investigated the hypothesis that this bottleneck can be overcome by simulated data. This talk will highlight some of the successes and challenges we encountered on our journey.