Clouds play an important role in balancing the Earth's energy budget. Research has indicated a rise in global average temperatures will lead to thinning of stratocumulus low clouds acting as a positive feedback on warming. Current state-of-the-art Earth System Models do not resolve cloud physics appropriately due to spatial resolution limitations, making it harder to model the cloud-climate feedback. In this study, we propose to learn this feedback with a transformer. To better respect the spatial structure of Earth, we transform the data to a spherical grid. Our resulting spherical transformer called ClimFormer -- using state of the art Fourier Neural Operator mixing -- is able to model this important energy exchange mechanism, and performs strongly on an out-of-distribution evaluation.