When faced with data-starved or highly complex end-tasks, it is commonplace for machine learning practitioners to introduce auxiliary objectives as supplementary learning signals. While much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand-design. Intuitions about how and when these objectives improve end-task performance have also had limited theoretical backing. In this talk I will present two works. First, I will discuss the widely used pre-train and fine-tune paradigm, and argue that when we know an end-task of interest before-hand we should also consider joint multi-task learning as a credible alternative. I will discuss an algorithm that we propose, META-TARTAN, that allows us to automatically learn the weights for the multi-task objective. Second, I will present AANG, an approach for automatically generating a suite of auxiliary objectives. AANG deconstructs existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task. We empirically verify that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP end-tasks.