Climate change poses new challenges to agricultural production, as crop yields are extremely sensitive to climatic variation. Accurately predicting the effects of weather patterns on crop yield is crucial for addressing issues such as food insecurity, supply stability, and economic planning. Recently, there have been many attempts to use machine learning models for crop yield prediction. However, these models either restrict their tasks to a relatively small region or a short time-period (e.g. a few years), which makes them hard to generalize spatially and temporally. They also view each location as an i.i.d sample, ignoring spatial correlations in the data. In this paper, we introduce a novel graph-based recurrent neural network for crop yield prediction, which incorporates both geographical and temporal structure. Our method is trained, validated, and tested on over 2000 counties from 41 states in the US mainland, covering years from 1981 to 2019. As far as we know, this is the first machine learning method that embeds geographical knowledge in crop yield prediction and predicts crop yields at the county level nationwide. Experimental results show that our proposed method consistently outperforms a wide variety of existing state-of-the-art methods, validating the effectiveness of geospatial and temporal information.