Climate and poverty are intrinsically related: regions with extreme temperatures, large temperature variability, and recurring extreme weather events tend to be ranked among the poorest and most vulnerable to climate change. Nevertheless, there currently is no established method to directly estimate the impact of specific climate variables on poverty and to identify geographical regions at high risk of being negatively affected by climate change. In this work, we propose a new approach based on Graph Neural Networks (GNNs) to estimate the effect of climate and remote sensing variables on poverty indicators measuring Education, Health, Living Standards, and Income. Furthermore, we use the trained models and perturbation analyses to identify the geographical regions most vulnerable to the potential variations in climate variables.