Skip to main navigation Skip to search Skip to main content

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

  • Zhengze Zhang
  • , Shiqi Wang
  • , Yiqun Shen
  • , Simin Guo
  • , Dahua Lin
  • , Xiaoliang Wang
  • , Cam Tu Nguyen*
  • , Fei Tan*
  • *Corresponding author for this work
  • Nanjing University
  • The University of Chicago
  • Chinese University of Hong Kong

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Large language models (LLMs) have demonstrated exceptional performance across various applications, but their conversational abilities decline sharply as model size decreases, presenting a barrier to their deployment in resource-constrained environments. Knowledge distillation with Direct Preference Optimization (dDPO) has emerged as a promising approach to enhancing the conversational abilities of smaller models using a larger teacher model. However, current methods primarily focus on “black-box” KD, which only uses the teacher's responses, overlooking the output distribution offered by the teacher. This paper addresses this gap by introducing daDPO (Distribution-Aware DPO), a unified method for preference optimization and distribution-based distillation. We provide rigorous theoretical analysis and empirical validation, showing that daDPO outperforms existing methods in restoring performance for pruned models and enhancing smaller LLM models. Notably, in in-domain evaluation, our method enables a 20% pruned Vicuna1.5-7B to achieve near-teacher performance (-7.3% preference rate compared to that of dDPO's -31%), and allows Qwen2.5-1.5B to occasionally outperform its 7B teacher model (14.0% win rate).

Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics
Subtitle of host publicationACL 2025
EditorsWanxiang Che, Joyce Nabende, Ekaterina Shutova, Mohammad Taher Pilehvar
PublisherAssociation for Computational Linguistics (ACL)
Pages15421-15437
Number of pages17
ISBN (Electronic)9798891762565
DOIs
StatePublished - 2025
Event63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 - Vienna, Austria
Duration: 27 Jul 2025 → 1 Aug 2025

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025
Country/TerritoryAustria
CityVienna
Period27/07/25 → 1/08/25

Fingerprint

Dive into the research topics of 'daDPO: Distribution-Aware DPO for Distilling Conversational Abilities'. Together they form a unique fingerprint.

Cite this