Utilizing computational methods to extract actional information from scientific data is essential due to the time-consuming and inaccurate nature of the manual processes of humans. To better serve the purpose, equipping computational methods with physical rules is necessary. Integrating deep learning models with symmetry awareness has emerged as a promising approach to significantly improve symmetry detection in experimental data, with techniques such as parameter sharing and novel convolutional layers enhancing symmetry recognition.[1,2,3,4,5,6] However, the challenge of integrating physical principles, such as symmetry, into these models persists. To address this, we have developed benchmarking datasets and training frameworks, exploring three perspectives to classify wallpaper group symmetries effectively. Our study demonstrates the limitations of deep learning models in understanding symmetry, as evidenced by benchmark results. A detailed analysis is provided with a hierarchical dataset and training outcomes, while a symmetry filter is designed aiming to improve symmetry operation recognition. This endeavor aims to push the boundaries of deep learning models in comprehending symmetry and embed physical rules within them, ultimately unlocking new possibilities at the intersection of machine learning and physical symmetry, with valuable applications in materials science and beyond.