TY - GEN
T1 - Improving Consistency Distillation with Rectified Trajectories
AU - Zhang, Han
AU - Cheng, Fan
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
PY - 2026
Y1 - 2026
N2 - Consistency models are a new family of generative models that can achieve high sample quality in one- and few-step generation. Its solid theoretical framework establishes it as the foundation for subsequent studies. Currently, a consistency model can be distilled from a pre-trained diffusion model through consistency distillation (CD). The utility of pre-trained models endows CD with a fast training convergence speed, thereby saving computational resources given the access to numerous pre-trained models in the thriving diffusion model community. However, current CD performs worse in sample quality than consistency training (CT) with improved techniques. To match the sample quality of CT while retaining CD’s ability to leverage pre-trained models, we present a novel distillation method dubbed improved consistency distillation (iCD). Specifically, our iCD integrate the unbiased Monte Carlo estimator used in CT with ordinary differential equation coupled data-noise pairs derived from the pre-trained diffusion model. Our insightful theoretical analysis indicates that this integration can simplify the trajectories that the consistency model learns. Benefiting from the simplified trajectories, our iCD can improve the sample quality of CD. Extensive experiments demonstrate that iCD inherits the fast training convergence of CD, while maintaining or even improving the sample quality of CT.
AB - Consistency models are a new family of generative models that can achieve high sample quality in one- and few-step generation. Its solid theoretical framework establishes it as the foundation for subsequent studies. Currently, a consistency model can be distilled from a pre-trained diffusion model through consistency distillation (CD). The utility of pre-trained models endows CD with a fast training convergence speed, thereby saving computational resources given the access to numerous pre-trained models in the thriving diffusion model community. However, current CD performs worse in sample quality than consistency training (CT) with improved techniques. To match the sample quality of CT while retaining CD’s ability to leverage pre-trained models, we present a novel distillation method dubbed improved consistency distillation (iCD). Specifically, our iCD integrate the unbiased Monte Carlo estimator used in CT with ordinary differential equation coupled data-noise pairs derived from the pre-trained diffusion model. Our insightful theoretical analysis indicates that this integration can simplify the trajectories that the consistency model learns. Benefiting from the simplified trajectories, our iCD can improve the sample quality of CD. Extensive experiments demonstrate that iCD inherits the fast training convergence of CD, while maintaining or even improving the sample quality of CT.
KW - Consistency models
KW - Diffusion models
KW - Rectified flow
UR - https://www.scopus.com/pages/publications/105016521040
U2 - 10.1007/978-3-032-04558-4_47
DO - 10.1007/978-3-032-04558-4_47
M3 - 会议稿件
AN - SCOPUS:105016521040
SN - 9783032045577
T3 - Lecture Notes in Computer Science
SP - 582
EP - 599
BT - Artificial Neural Networks and Machine Learning – ICANN 2025 - 34th International Conference on Artificial Neural Networks, 2025, Proceedings
A2 - Senn, Walter
A2 - Sanguineti, Marcello
A2 - Saudargiene, Ausra
A2 - Tetko, Igor V.
A2 - Villa, Alessandro E. P.
A2 - Jirsa, Viktor
A2 - Bengio, Yoshua
PB - Springer Science and Business Media Deutschland GmbH
T2 - 34th International Conference on Artificial Neural Networks, ICANN 2025
Y2 - 9 September 2025 through 12 September 2025
ER -