Skip to yearly menu bar Skip to main content


Poster
in
Workshop: NeurIPS 2023 Workshop on Diffusion Models

Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization

Yinbin Han · Meisam Razaviyayn · Renyuan Xu


Abstract:

Diffusion models, rivaling GANs for high-quality sample generation, leverage score matching to learn the score function. Despite empirical success, the provable accuracy of gradient-based algorithms for this task remains unclear. As a first step toward answering this question, this paper establishes a mathematical framework for analyzing score estimation using neural networks trained by gradient descent. The analysis covers both the optimization and the generalization aspects of the training procedure. We propose a parametric form to formulate the denoising score-matching problem as a regression with noisy labels. Compared to standard supervised learning, the score-matching problem introduces distinct challenges, including unbounded inputs, vector-valued outputs, and an additional time variable, preventing existing techniques from being applied directly. With a properly designed neural network architecture, we demonstrate an accurate approximation of the score function using a reproducing kernel Hilbert space induced by neural tangent kernels. We establish the first generalization error bound for learning the score function by applying early stopping and coupling arguments.

Chat is not available.