A desirable feature of interactive NLP systems is the ability to receive feedback from humans and personalize to new users. Existing paradigms encounter challenges in acquiring new concepts due to the use of discrete labels and scalar rewards. As one solution to alleviate this problem, I will present our work on Semantic Supervision (SemSUP), which trains models to predict over multiple natural language descriptions of classes (or even structured ones like JSON). SemSUP can seamlessly replace any standard supervised learning setup without sacrificing any in-distribution accuracy, while providing generalization to unseen concepts and scalability to large label spaces.