Curriculum learning has been a quiet yet crucial component of many of the high-profile successes of reinforcement learning. Despite this, none of the major reinforcement learning libraries support curriculum learning or include curriculum learning algorithms. Curriculum learning methods can provide general and complementary improvements to RL algorithms, but they often require significant, complex changes to agent training code. We introduce Syllabus, a library for training RL agents with curriculum learning, as a solution to this problem. Syllabus provides a universal API for implementing curriculum learning algorithms, a collection of implementations of popular curriculum learning methods, and infrastructure for easily integrating them into existing distributed RL code. Syllabus provides a clean API for each of the complex components of these methods, dramatically simplifying the process for designing new algorithms or applying existing algorithms to new environments. Syllabus also manages the multiprocessing communication required for curriculum learning, alleviating one of the key practical challenges of using these algorithms. We hope Syllabus will improve the process of developing and applying curriculum learning algorithms, and encourage widespread adaptation of curriculum learning.