Data-driven weather and climate forecasts on subseasonal-to-seasonal (S2S) scales face a two-fold challenge: the observational records are comparatively short and predictability strongly depends on the coupling of earth systems evolving on very different timescales.As a first step towards general subseasonal-to-seasonal predictions, we model the El Niño Southern Oscillation (ENSO), one of the principal sources of predictability on S2S scales.Characterised by anomalously warm sea surface temperatures in the tropical Pacific, an El Niño event arises from a combination of slow heat transfer within the Pacific and between ocean basins, as well as fast atmospheric dynamics, such as westerly wind bursts and convection. Here, we design a deep learning model that can flexibly represent information from modalities on different timescales, trained as a Masked Autoencoder and optimising the Empirical Continuous Ranked Probability Score. We find that the representation learning approach exhibits zero-shot performance that is competitive with task-specific models on S2S of ENSO in terms of correlation skill.To the best of our knowledge, we are the first to predict well-calibrated uncertainty estimates on a 24 month horizon.