As autonomous decision-making agents move from narrow operating environments to unstructured worlds, learning systems must move from a closed-world formulation to an open-world, incremental, few-shot setting in which agents continuously learn new labels from small amounts of information. This stands in stark contrast to modern machine learning systems that are typically designed with a known set of classes and a large number of examples for each class. In this work, we extend embedding-based few-shot learning algorithms toward open-world problems. In particular, we investigate both the lifelong setting---in which an entirely new set of classes exists at evaluation time---as well as the incremental setting, in which new classes are added to a set of base classes available at training time. We combine Bayesian non-parametric class priors with an embedding-based pre-training scheme to yield a highly flexible framework for use in both the lifelong and the incremental settings. We benchmark our framework on MiniImageNet and, and show strong performance compared to baseline methods.