Encoding and predicting physical measurements such as temperature or carbon dioxide is instrumental to many high-stakes challenges – including climate change. Yet, all recent advances solely assess models’ performances at a global scale. But while models’ predictions are improving on average over the entire globe, performances on sub-groups such as islands or coastal areas are left uncharted. To ensure safe deployment of those models, we thus introduce FAIR-Earth, a fine-grained evaluation suite made of diverse and high-resolution dataset. Our findings are striking–current methods produce highly biased predictions towards specific geospatial locations. The specifics of the biases vary based on the data modality and hyper-parameters of the models. Hence, we hope that FAIR-Earthwill enable future research to design solutions aware of those per-group biases.