A spectrogram is a visual representation of the spectrum of frequencies in a signal as it varies with time.An important problem in radio astronomy is removing radio frequency interference (RFI) from the spectrogram produced by radio telescopes. Given the competitive performance of diffusion models image in-painting, and the similarity of spectrogram to image data, such methods seem like a natural choice for correcting the corruption due to RFI. Unfortunately, this application is complicated because the entire dataset is corrupted; therefore, existing solutions relying on a clean training set are not readily applicable. Moreover, we observethat in contrast to image data, spectrograms do not have translation symmetry along the frequency axis undermines the prevalent use of ConvNets in processing this kind of data.Fortunately, the RFI corruptions are often local in nature (in time or frequency), meaning we can easily identify and mask the RFI region. In this paper, we investigate a progressively improving series of solutions to the problem of image in-painting, where all the training data is corrupted through masking. Moreover, we propose a positional encoding scheme to break the translation symmetry assumed by ConvNets.We experiment with the CIFAR10 dataset and synthetic spectrogram data, where our initial results are quite supportive.