Solving image inverse problems (e.g., super-resolution and inpainting) requires generating a high fidelity image that matches the given input (the low-resolution image or the masked image). By using the input image as guidance, we can leverage a pretrained diffusion generation model to solve a wide range of image inversion tasks without task specific model fine-tuning. In this work, we propose diffusion policy gradient (DPG), a tractable computation method to estimate the score function given the guidance image. Our method is robust to both Gaussian and Poisson noise added to the input image, and it improves the image restoration consistency and quality on FFHQ, ImageNet and LSUN datasets on both linear and non-linear image inversion tasks (inpainting, super-resolution, motion deblur, non-linear deblur, etc.).