Annotating cancerous regions in whole-slide images (WSIs) plays a critical role in clinical diagnosis and biomedical research, but generating such exhaustive and accurate annotations is labor-intensive and costly. We present a method, named Label Cleaning Multiple Instance Learning (LC-MIL), to refine coarse and approximate annotations – much easier and less costly to obtain – to produce more accurate ones, on a single WSI without the need of external training data. Our experiments on a heterogeneous set of diverse cancer types demonstrate that LC-MIL is a promising and light-weight tool to provide fine-grained and accurate annotations from coarsely annotated pathology sets.