Skip to main navigation Skip to search Skip to main content

Improving Consistency Distillation with Rectified Trajectories

  • Han Zhang
  • , Fan Cheng*
  • *Corresponding author for this work
  • Shanghai Jiao Tong University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Consistency models are a new family of generative models that can achieve high sample quality in one- and few-step generation. Its solid theoretical framework establishes it as the foundation for subsequent studies. Currently, a consistency model can be distilled from a pre-trained diffusion model through consistency distillation (CD). The utility of pre-trained models endows CD with a fast training convergence speed, thereby saving computational resources given the access to numerous pre-trained models in the thriving diffusion model community. However, current CD performs worse in sample quality than consistency training (CT) with improved techniques. To match the sample quality of CT while retaining CD’s ability to leverage pre-trained models, we present a novel distillation method dubbed improved consistency distillation (iCD). Specifically, our iCD integrate the unbiased Monte Carlo estimator used in CT with ordinary differential equation coupled data-noise pairs derived from the pre-trained diffusion model. Our insightful theoretical analysis indicates that this integration can simplify the trajectories that the consistency model learns. Benefiting from the simplified trajectories, our iCD can improve the sample quality of CD. Extensive experiments demonstrate that iCD inherits the fast training convergence of CD, while maintaining or even improving the sample quality of CT.

Original languageEnglish
Title of host publicationArtificial Neural Networks and Machine Learning – ICANN 2025 - 34th International Conference on Artificial Neural Networks, 2025, Proceedings
EditorsWalter Senn, Marcello Sanguineti, Ausra Saudargiene, Igor V. Tetko, Alessandro E. P. Villa, Viktor Jirsa, Yoshua Bengio
PublisherSpringer Science and Business Media Deutschland GmbH
Pages582-599
Number of pages18
ISBN (Print)9783032045577
DOIs
StatePublished - 2026
Externally publishedYes
Event34th International Conference on Artificial Neural Networks, ICANN 2025 - Kaunas, Lithuania
Duration: 9 Sep 2025 → 12 Sep 2025

Publication series

NameLecture Notes in Computer Science
Volume16068 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference34th International Conference on Artificial Neural Networks, ICANN 2025
Country/TerritoryLithuania
CityKaunas
Period9/09/25 → 12/09/25

Keywords

  • Consistency models
  • Diffusion models
  • Rectified flow

Fingerprint

Dive into the research topics of 'Improving Consistency Distillation with Rectified Trajectories'. Together they form a unique fingerprint.

Cite this