In a typical robot manipulation setting, the physical laws that govern object dynamics never change, but the set of objects does. To complicate matters, objects may have intrinsic properties that are not directly observable (e.g., center of mass or friction coefficients). In this work, we introduce a latent-variable model of object-factored dynamics. This model represents uncertainty about the dynamics using deep ensembles while capturing uncertainty about each object's intrinsic properties using object-specific latent variables. We show that this model allows a robot to rapidly generalize to new objects by using information theoretic active learning. Additionally, we highlight the benefits of the deep ensemble for robust performance in downstream tasks.