跳到主要导航 跳到搜索 跳到主要内容

Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting

  • Zeyuan Chen
  • , Haiyan Wu
  • , Kaixin Wu
  • , Wei Chen
  • , Mingjie Zhong
  • , Jia Xu*
  • , Zhongyi Liu
  • , Wei Zhang*
  • *此作品的通讯作者
  • Ant Group
  • Alibaba Group Holding Ltd.

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

Relevance modeling is a critical component for enhancing user experience in search engines, with the primary objective of identifying items that align with users’ queries. Traditional models only rely on the semantic congruence between queries and items to ascertain relevance. However, this approach represents merely one aspect of the relevance judgement, and is insufficient in isolation. Even powerful Large Language Models (LLMs) still cannot accurately judge the relevance of a query and an item from a semantic perspective. To augment LLMs-driven relevance modeling, this study proposes leveraging user interactions recorded in search logs to yield insights into users’ implicit search intentions. The challenge lies in the effective prompting of LLMs to capture dynamic search intentions, which poses several obstacles in real-world relevance scenarios, i.e., the absence of domain-specific knowledge, the inadequacy of an isolated prompt, and the prohibitive costs associated with deploying LLMs. In response, we propose ProRBP, a novel Progressive Retrieved Behavior-augmented Prompting framework for integrating search scenario-oriented knowledge with LLMs effectively. Specifically, we perform the user-driven behavior neighbors retrieval from the daily search logs to obtain domain-specific knowledge in time, retrieving candidates that users consider to meet their expectations. Then, we guide LLMs for relevance modeling by employing advanced prompting techniques that progressively improve the outputs of the LLMs, followed by a progressive aggregation with comprehensive consideration of diverse aspects. For online serving, we have developed an industrial application framework tailored for the deployment of LLMs in relevance modeling. Experiments on real-world industry data and online A/B testing demonstrate our proposal achieves promising performance.

源语言英语
主期刊名Industry Track
编辑Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Darwish, Apoorv Agarwal
出版商Association for Computational Linguistics (ACL)
页784-793
页数10
ISBN(电子版)9798891761971
出版状态已出版 - 2025
活动31st International Conference on Computational Linguistics, COLING 2025 - Abu Dhabi, 阿拉伯联合酋长国
期限: 19 1月 2025 → 24 1月 2025

丛书

姓名Proceedings - International Conference on Computational Linguistics, COLING
ISSN(印刷版)2951-2093

会议

会议31st International Conference on Computational Linguistics, COLING 2025
国家/地区阿拉伯联合酋长国
市Abu Dhabi
时期19/01/25 → 24/01/25

学术指纹

探究 'Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting' 的科研主题。它们共同构成独一无二的学术指纹。

引用此