Technological advances in medical data collection such as high-resolution histopathology and high-throughput genomic sequencing have contributed to the possibility to contextualise computer vision models with genomic information in a multi-modal manner. Complementing imaging models with other modalities has shown promise in providing cell-, molecular-, and patient-level context to improve overall predictive performance. However, the context representations for other modalities are often learned with modality-specific encoders, which cannot capture the crucial cross-modal information that motivates the integration of different data sources. This paper presents a Hybrid Early-fusion Attention Learning Network (HEALNet) – a flexible multi-modal fusion architecture that learns both a shared and modality-specific parameter space that a) preserves modality-specific structural information, b) captures the cross-modal interactions and structural information in a shared latent space, c) effectively handles missing modalities during training and inference, and d) enables intuitive model inspection by learning on the raw data input instead of opaque embeddings. We conduct multi-modal survival analysis on Whole Slide Images and Multi-omic data on four cancer cohorts of The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-the-art performance, substantially improving over both image-only and recent multi-modal baselines, whilst being robust in scenarios with missing modalities.