Spotlight Poster
Discrete Flow Matching
Itai Gat · Tal Remez · Neta Shaul · Felix Kreuk · Ricky T. Q. Chen · Gabriel Synnaeve · Yossi Adi · Yaron Lipman
[
Abstract
]
Fri 13 Dec 11 a.m. PST
— 2 p.m. PST
Abstract:
Flow Matching and diffusion models have emerged as powerful generative paradigms for continuous variables such as images and videos. However, their application to high-dimensional discrete data, such as language, has been limited. In this work, we present Discrete Flow Matching, a novel discrete flow paradigm designed specifically for generating discrete data. Discrete Flow Matching offers several key contributions: (i) it works with a general family of probability paths interpolating between source and target distributions; (ii) it allows for a generic formula for sampling from these probability paths using learned posteriors such as the denoiser ($x$-prediction) and noise-prediction ($\epsilon$-prediction); (iii) practically, focusing on specific probability paths defined with different schedulers considerably improves generative perplexity compared to previous discrete diffusion and flow models; and (iv) scaling Discrete Flow Matching models up to 1.7B parameters, we reach 6.7\%\,Pass@1 and 13.4\%\,Pass@10 on HumanEval and 6.7\%\,Pass@1 and 20.6\%\,Pass@10 on 1-shot MBPP coding benchmarks. Our approach is capable of generating high-quality discrete data in a non-autoregressive fashion, significantly closing the gap between autoregressive models and discrete flow models.
Live content is unavailable. Log in and register to view live content