If AGI is right around the corner, why are embodied AI agents deployed in real-world settings still so dumb? In open world settings, AI still fails to coordinate effectively with other agents, follow natural language instructions to complete embodied tasks, and generalize to circumstances not encountered during training. In contrast, humans and animals can easily adapt to new circumstances, coordinate with others, and acquire complex behaviors. I argue that Social Learning is a key facet of intelligence that gives rise to all of these impressive capabilities. By improving the social intelligence of AI agents, we can get a step closer to adaptive, flexible, generalist open world agents. This talk will overview recent work in the Social Reinforcement Learning lab, describing how to enable smooth coordination with diverse human partners, improve social reasoning for understanding natural language commands, and use social learning to enable rapid online adaptation to new environments by learning from experts.