Recently there is an increasing interest in learning to simulate the dynamics of physic systems via machine learning. However, existing approaches fail to generalize to physical substances not in the training set, such as liquids with different viscosities or elastomers with different elasticities. Here we present a machine learning method embedded with physical priors and material parameters, which we term as “Graph-based Physics Engine” (GPE), to efficiently model the physical dynamics of different substances in a wide variety of challenging scenarios. We demonstrate that GPE can generalize to different material properties not seen in the training set by simply modifying the physical parameters, and also performs well from single-step predictions to multi-step roll-out simulations. GPE provides new insights into the construction of learnable simulators and is a key step toward predicting unknown physics problems in the real world.