Machine learning is increasingly successful in modeling physical systems in science and engineering. Integrating physical information, like PDEs, can enhance model performance and address generalization issues caused by limited, costly data. However, since PDEs heavily rely on fine-grained states to calculate the derivative, the integration of physical information into models is significantly challenged by the coarse-grained nature of measurement, often due to sensor limitations. To address the challenge, we introduce the Physics-Informed Coarse-grained data Learning (PICL) framework to enhance the model's generalization for predicting future coarse-grained observations. The key idea is to re-enable the physics-based loss on the transition between adjacent fine-grained states corresponding to the available coarse-grained data. The challenge is how to reconstruct the corresponding fine-grained state by only using coarse-grained data. We discover that the physics-based loss can also address this challenge. PICL combines an encoding module for reconstructing learnable fine-grained states with a transition module for predicting future states. The two modules are jointly trained by data and physical information. The experiment results confirm PICL's significant improvement in generalization capacity in various physical systems.