Embodied intelligence is the ultimate lifelong learning problem. If you had a robot in your home, you would likely ask it to do all sorts of varied chores, like setting the table for dinner, preparing lunch, and doing a load of laundry. The things you would ask it to do might also change over time, for example to use new appliances. You would want your robot to learn to do your chores and adapt to any changes quickly. In this talk, I will explain how we can leverage various forms of modularity that are common in robotics to develop powerful lifelong learning mechanisms. My talk will dive into some algorithms that leverage this key insight in the contexts of reinforcement learning and model-based planning. I will then describe my vision for how we can create the next generation of home assistant robots that leverage large-scale data to continually improve their own capabilities.