Optical multilayer thin film structures are widely used in various photonic applications. Inverse design is an important but difficult step to enable these applications, which seeks to find out the best structure (material & thickness arrangements) given a target optical response. Recently, deep learning-based methods have been developed to solve the inverse design efficiently. However, existing methods usually fix the material arrangements and only design the thickness, which is not versatile for a different material arrangement and may lead to sub-optimal performance. In this study, we resolve this issue by treating the structure as a sequence and using structure tokens to represent the material and thickness simultaneously. Later on, the inverse design problem can be formulated as a common sequence generation task conditioned on the input optical responses. Based on this, we propose OptoGPT to act as a versatile inverse design model that can design material and thickness simultaneously, significantly expanding the design capability. In addition, using probability resampling further provides a versatile method to satisfy fabrication and design requirements in practical applications.