Skip to yearly menu bar Skip to main content


Poster
in
Workshop: NeurIPS 2023 Workshop on Diffusion Models

Improving Discrete Diffusion Models via Structured Preferential Generation

Severi Rissanen · Markus Heinonen · Arno Solin


Abstract:

In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores. This work paves the way for more advanced discrete diffusion models with potentially significant enhancements in performance.

Chat is not available.