While graph neural networks (GNNs) provide a powerful way to learn structured representations, it remains challenging to learn long-range dependencies between graph nodes. Recurrent gated GNNs only partly address this problem. We introduce a memory augmentation to a gated GNN which simply stores the previous hidden states in a cache. We show that the cache-memory gated GNN outperforms other models on a synthetic task that requires long-range information, as well as tasks on real-world datasets.