Constructing a robust model that can effectively generalize to test samples under distribution shifts remains a significant challenge in the field of medical imaging. The vision-language foundation model has recently emerged as a promising paradigm, demonstrating impressive learning capabilities across various tasks while requiring a small amount of finetuning samples. While numerous approaches have focused on developing better fine-tuning strategies for specific domains, we instead examine the robustness of such foundation models to domain shifts in the medical image segmentation task. To this end, we compare the generalization performance to unseen domains of various pre-trained models after being finetuned on the same in-distribution dataset and show that foundation-based models enjoy better robustness compared to other architectures. From here, we further developed a new Bayesian uncertainty estimation for frozen models and used them as an indicator to characterize the model's performance on out-of-distribution (OOD) data, which can be extremely useful for real-world applications. Our experiments show the shortcomings of existing indicators used in natural image applications and the promising results of the proposed Bayesian uncertainty.