Designing free-form photonic devices is a challenging topic due to the high degree of structural freedom. Here, we present \textit{AdjointDiffusion}, a new algorithm that optimizes photonic structures using adjoint sensitivity analysis and diffusion models. We demonstrate that integrating adjoint gradient values into the denoising process enables the generation of high-performance device structures. Our method can optimize structures with a small number of simulations by incorporating a diffusion model trained on synthetic images that follow fabrication constraints. Compared to conventional algorithms, our approach eliminates the need for intricate binarization and conic filters, overcomes the issue of local optima, and provides a variety of design options. Despite the inherent stochasticity, our algorithm robustly designs high-performance devices and outperforms state-of-the-art nonlinear algorithms.