Topological defects are omnipresent in physical and biological systems. These localized deforming structures have an intrinsic multiscale nature: Their microscopic structure can be represented macroscopically as interacting particles. In this work, we extend a Wasserstein Generative Adversarial Network (WGAN) by incorporating physical constraints, to reconstruct realistic microscopic structures from macroscopic topological defect distributions and other physical inputs. Using the two-dimensional XY-model as proof of concept, our method generates physical realistic micro-magnetic vector configurations verified through multiple measures. This approach might allow efficient dynamical simulations of large defect systems and support analytical methods of experimental data, as microscale information can be recovered using our network.