In neuroimaging, accurate brain age prediction is key to understanding brain aging and early neurodegenerative signs. Recent advancements in self-supervised learning, particularly contrastive learning, have shown robustness with complex datasets but struggle with non-uniformly distributed data common in medical imaging. We introduce a novel contrastive loss that dynamically adapts during training, focusing on localized sample neighborhoods. Additionally, we incorporate brain stiffness, a mechanical property sensitive to aging. Our approach outperforms state-of-the-art methods and opens new directions for brain aging research.