Text-to-image Generation
LlamaGen (256-step generation)
t2ibaseline
Distilled Decoding (DD) (2-step generation, 93 speedup)
t2idd
Generation on ImageNet-256
LlamaGen (256-step generation)
baseline
Distilled Decoding (DD) (2-step generation, 118 speedup)
dd2step
Distilled Decoding (DD) (1-step generation, 217 speedup)
dd1step

VAR (10-step generation)
baseline
Distilled Decoding (DD) (2-step generation, 3.7 speedup)
dd2step
Distilled Decoding (DD) (1-step generation, 6.3 speedup)
dd1step

Abstract

Autoregressive (AR) models have recently achieved state-of-the-art performance in text and image generation. However, their primary limitation is slow generation speed due to the token-by-token process. We ask an ambitious question: can a pre-trained AR model be adapted to generate outputs in just one or two steps? If successful, this would significantly advance the development and deployment of AR models. We notice that existing works that attempt to speed up AR generation by generating multiple tokens at once fundamentally cannot capture the output distribution due to the conditional dependencies between tokens, limiting their effectiveness for few-step generation. To overcome this, we propose Distilled Decoding (DD), which leverages flow matching to create a deterministic mapping from Gaussian distribution to the output distribution of the pre-trained AR model. We then train a network to distill this mapping, enabling few-step generation. The entire training process of DD does not need the training data of the original AR model (as opposed to some other methods), thus making DD more practical. We evaluate DD on state-of-the-art image AR models and present promising results. For VAR, which requires 10-step generation (680 tokens), DD enables one-step generation (6.3 speed-up), with an acceptable increase in FID from 4.19 to 9.96 on ImageNet-256. Similarly, for LlamaGen, DD reduces generation from 256 steps to 1, achieving an 217.8 speed-up with a comparable FID increase from 4.11 to 11.35 on ImageNet-256. In both cases, baseline methods completely fail with FID scores >100. DD also excels on text-to-image generation, reducing the generation from 256 steps to 2 for LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to demonstrate the possibility of one-step generation for image AR models, DD challenges the prevailing notion that AR models are inherently slow, and opens up new opportunities for efficient AR generation.

Highlight

 

Quantitative Results

Main Results
Main Results on ImageNet-256.
AR Flow Matching
Results of teacher-involved sampling on ImageNet-256. The notation pre-trained-n-m means that the pre-trained AR model is used to re-generate n-th to m-1-th tokens.
T2I Results
Results of text-to-image task on LAION-COCO. FIDs are caculated with 5k images.

BibTeX

@article{liu2024distilleddecoding1onestep, title={Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching}, author={Enshu Liu and Xuefei Ning and Yu Wang and Zinan Lin}, year={2024}, journal={arXiv preprint arXiv:2412.17153}, }