MCFC: A Momentum-Driven Clicked Feature Compressed Pre-trained Language Model for Information Retrieval

Information Retrieval (IR) pre-trained language models are trained from large-scale retrieval-based corpora to promote the task-specific knowledge capacity. Previous works focus on general retrieval pre-trained datasets, which cover inter-document data and intra-document data, paying less attention to the important asset of clicked data which is commonly adopted in recommendation domain. However, the utilization of easily accessible clicked data is a non-trivial operation due to its characteristics of large volume and insufficient refinement, which affect model learning efficiency and imply the risk of distorting learning directions. In this paper, we propose a Momentum-Driven Clicked Feature Compressed Pre-trained Language Models for Information Retrieval (MCFC). Specifically, to tackle the effective learning pace on large amounts of data, we generalize multiple similar feature instances and compress the dispersed knowledge together at the query granularity, named Multi-Instance Information Integration. Meanwhile, more relevant detection between queries and documents is eager in coarse clicked data background, we leverage a momentum-driven adjusting mechanism to refine the text representations, named Continuous Debiasing Calibration. Extensive experiments on downstream datasets validate the superiority of our work to other recent strong baselines.