Optical multilayer thin film structures are widely used in many photonic applications, including filters, absorbers, photovoltaics, display devices. The important part to enable these applications is inverse design, which seeks to identify a suitable structure that satisfy desired optical responses. Recently, a foundation model called OptoGPT is proposed and has shown great potential to solve a wide range of inverse design problems. However, OptoGPT fails to design certain types of optical responses that are important to practical applications. The major reason is that the training data is randomly sampled and it is possible these design targets are not selected in training, leading to the out-of-distribution issue. In this work, we propose a self-improving data augmentation technique by leveraging neural networks' extrapolation ability. Using this method, we show significant improvement in various real-applicative design tasks with minimum fine-tuning, which can also be potentially generalized to inverse scientific foundation models.