Two-dimensional nanomaterials, such as graphene, have been extensively studied because of their outstanding physical properties. Structure and geometry optimization of nanopores on such materials is beneficial for their performance in real-world engineering applications such as water desalination. However, the optimization process often involves very large numbers of experiments or simulations which are expensive and time-consuming. In this work, we propose a graphene nanopore optimization framework via the combination of deep reinforcement learning (DRL) and convolutional neural network (CNN) for efficient water desalination. The DRL agent controls the geometry of nanopore, while the CNN is employed to predict the water flux and ion rejection of the nanoporous graphene membrane at a certain external pressure. With the CNN-accelerated property prediction, our DRL agent can optimize the nanoporous graphene efficiently in an online manner. Experiments show that our framework can design nanopore structures that are promising in energy-efficient water desalination.