There have been many recent advances in leveraging machine learning for chemistry applications. One particular task of interest is using graph neural networks (GNNs) on the Open Catalyst 2020 (OC20) dataset to predict the forces and energies of atoms and systems. While large GNNs have shown good progress in this area, we have little understanding of how or why these models work. In an attempt to gain a better understanding and increase our confidence that the models learn meaningful concepts that align with chemical intuition, we present perturbation analyses of GNN predictions on OC20, where we performed small changes on individual atoms and compared the model predictions before and after the changes. We provide visualizations of individual systems as well as analyses on general trends. We observed evidence that aligns with chemical intuition, including the importance of adsorbate atoms on the overall system, that modifying atomic numbers to neighbors of the same row of the periodic table causes less difference than other elemental changes, and a positive correlation between force magnitudes and energy changes.