We present an equivariant neural network for predicting the phonon modes of the periodic crystals and molecules by evaluating the second derivative Hessian matrices of the energy model, which are first trained with the energy and force data. Such efficient Hessian prediction enables us to predict the phonon dispersion and the density of states for inorganic crystal material and can be fine-tuned with additional dataset. For molecules, we also derive the symmetry constraints for infrared/Raman active modes by analyzing the phonon mode irreducible representations. Our training paradigm further shows using Hessian as a new type of higher-order training data to improve the energy models beyond the lower-order energy and force data.