Heterogeneity in medical imaging data is often tackled, in the context of machine learning, using domain invariance, i.e. deriving models that are robust to domain shifts, which can be both within domain (e.g. demographics) and across domains (e.g. scanner/protocol characteristics). However this approach can be detrimental to performance because it necessitates averaging across intra-class variability and reduces discriminatory power of learned models, in order to achieve better intra- and inter-domain generalization. This paper instead embraces the heterogeneity and treats it as a multi-task learning problem to explicitly adapt trained classifiers to both inter-site and intra-site heterogeneity. We demonstrate that the error of a base classifier on challenging 3D brain magnetic resonance imaging (MRI) datasets can be reduced by 2-3x, in certain tasks, by adapting to the specific demographics of the patients, and different acquisition protocols. Learning the characteristics of domain shifts is achieved via auxiliary learning tasks leveraging commonly available data and variables, e.g. demographics. In our experiments, we use gender classification and age regression as auxiliary tasks helping the network weights trained on a source site adapt to data from a target site; we show that this approach improves classification error by 5-30% across different datasets on the main classification tasks, e.g. disease classification.