Skip to main navigation Skip to search Skip to main content

Motion-aware Contrastive Video Representation Learning via Foreground-background Merging

  • Shuangrui Ding
  • , Maomao Li
  • , Tianyu Yang
  • , Rui Qian
  • , Haohang Xu
  • , Qingyi Chen
  • , Jue Wang
  • , Hongkai Xiong*
  • *Corresponding author for this work
  • Shanghai Jiao Tong University
  • Tencent
  • Chinese University of Hong Kong
  • University of Michigan, Ann Arbor

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In light of the success of contrastive learning in the image domain, current self-supervised video representation learning methods usually employ contrastive loss to facilitate video representation learning. When naively pulling two augmented views of a video closer, the model however tends to learn the common static background as a shortcut but fails to capture the motion information, a phenomenon dubbed as background bias. Such bias makes the model suffer from weak generalization ability, leading to worse performance on downstream tasks such as action recognition. To alleviate such bias, we propose Foreground-background Merging (FAME) to deliberately compose the moving foreground region of the selected video onto the static background of others. Specifically, without any off-the-shelf detector, we extract the moving fore-ground out of background regions via the frame difference and color statistics, and shuffle the background regions among the videos. By leveraging the semantic consistency between the original clips and the fused ones, the model focuses more on the motion patterns and is debiased from the background shortcut. Extensive experiments demonstrate that FAME can effectively resist background cheating and thus achieve the state-of-the-art performance on downstream tasks across UCF101, HMDB51, and Diving48 datasets. The code and configurations are released at https://github.com/Mark12Ding/FAME.

Original languageEnglish
Title of host publicationProceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
PublisherIEEE Computer Society
Pages9706-9716
Number of pages11
ISBN (Electronic)9781665469463
DOIs
StatePublished - 2022
Externally publishedYes
Event2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022 - New Orleans, United States
Duration: 19 Jun 2022 → 24 Jun 2022

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Volume2022-June
ISSN (Print)1063-6919

Conference

Conference2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Country/TerritoryUnited States
CityNew Orleans
Period19/06/22 → 24/06/22

Keywords

  • Self-& semi-& meta- Representation learning

Fingerprint

Dive into the research topics of 'Motion-aware Contrastive Video Representation Learning via Foreground-background Merging'. Together they form a unique fingerprint.

Cite this