We introduce ViXReg, a framework that adapts Vision Transformers (such as Google ViT, Swin, BEiT) to tackle image analysis challenges in Inertial Confinement Fusion (ICF) radiography. ViXReg works by repurposing pixel-level classification capabilities for advanced image regression tasks, effectively reconstructing asymmetric double-shell structures—crucial for diagnosing nuclear fusion dynamics and capturing instabilities in high-energy-density plasmas under extreme thermal conditions. Our investigation explores architectural adaptations, including nonlinear and linear mappings, and advanced fine-tuning strategies like multi-scale pre-training and knowledge distillation, enhancing model scalability and generalization across diverse data distributions. Evaluating 60,000 synthetic ICF radiographs and 115 radiographs captured from 6 ICF experimental shots, we further craft domain adaptation techniques with weakly pseudo-labeled data, enabling ViXReg to transfer robust representations effectively to experimental dataset. The results from the above tasks demonstrate considerable advancements in using transformers as backbone architectures for fusion imagery, effectively capturing the subtle double-shell structures identified in plasma physics. Additionally, fine-tuning the pre-trained ViXReg model accelerates training convergence and enhances the accuracy of double-shell reconstructions, surpassing the performance of traditional convolutional neural networks and generative adversarial models. These findings demonstrate ViXReg's potential as a candidate for a foundation model component in scientific modeling for nuclear fusion research.