In medical data analysis, human practitioners have long excelled at adopting a holistic approach, considering a wide range of patient information, including multiple imaging sources and evolving medical histories. A prime example can be found in tumor boards, where, among others, radiologists evaluate a multitude of images while taking into account dynamic patient narratives. However, within the domain of medical image analysis, the current focus often narrows down to individual images, or if longitudinal data is used, the task is to infer non-dense predictions, like classification.However, if we can leverage multiple time points and model a patient trajectory, we can predict patient status on an image level at an arbitrary time point in the future.It could not only lead to better predictions for the current time point, i.e. for image segmentation, but it can also lead to a more holistic approach in medical image analysis, more akin to the human approach.In response to this disparity, our work, we motivate the need for longitudinal medical image analysis and we present a model that can deal with sparse and irregular longitudinal series, and without sacrificing generality, generate images.