Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling

Pan Chen; Shaohui Lin; Yongxiang Zhang; Jiachen Xu; Xin Tan; Lizhuang Ma

doi:10.1007/978-3-030-92273-3_20

Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling

Pan Chen
, Shaohui Lin
, Yongxiang Zhang
, Jiachen Xu
, Xin Tan
, Lizhuang Ma^*

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Compressed video action recognition targets at classifying action class in compressed video, instead of decoded/standard video. It benefits from fast training and inference by reducing the utilization of redundant information. However, off-the-shelf methods still rely on heavy-cost labels for training. In this paper, we propose self-supervised compressed video action recognition method via Momentum contrast (MoCo) and temporal-consistent sampling. We leverage temporal-consistent sampling into MoCo to improve the ability of feature presentation on each input modality of compressed video. Modality-oriented fine-tuning is introduced to applying into the downstream compressed video action recognition. Extensive experiments demonstrate the effectiveness of our method on different datasets with different backbones. Compared to SOTA self-supervised learning methods for decoded videos on HMDB51 dataset, our method achieves the highest accuracy of 57.8%.

Original language	English
Title of host publication	Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings
Editors	Teddy Mantoro, Minho Lee, Media Anugerah Ayu, Kok Wai Wong, Achmad Nizar Hidayanto
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	237-249
Number of pages	13
ISBN (Print)	9783030922726
DOIs	https://doi.org/10.1007/978-3-030-92273-3_20
State	Published - 2021
Event	28th International Conference on Neural Information Processing, ICONIP 2021 - Virtual, Online Duration: 8 Dec 2021 → 12 Dec 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13111 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	28th International Conference on Neural Information Processing, ICONIP 2021
City	Virtual, Online
Period	8/12/21 → 12/12/21

Keywords

Action recognition
Compressed video
Contrastive learning
Temporal-consistent sampling

Access to Document

10.1007/978-3-030-92273-3_20

Cite this

Chen, P., Lin, S., Zhang, Y., Xu, J., Tan, X., & Ma, L. (2021). Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling. In T. Mantoro, M. Lee, M. A. Ayu, K. W. Wong, & A. N. Hidayanto (Eds.), Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings (pp. 237-249). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13111 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-92273-3_20

Chen, Pan ; Lin, Shaohui ; Zhang, Yongxiang et al. / Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling. Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings. editor / Teddy Mantoro ; Minho Lee ; Media Anugerah Ayu ; Kok Wai Wong ; Achmad Nizar Hidayanto. Springer Science and Business Media Deutschland GmbH, 2021. pp. 237-249 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{856c36e6f9f6473eaf365982c5464535,

title = "Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling",

abstract = "Compressed video action recognition targets at classifying action class in compressed video, instead of decoded/standard video. It benefits from fast training and inference by reducing the utilization of redundant information. However, off-the-shelf methods still rely on heavy-cost labels for training. In this paper, we propose self-supervised compressed video action recognition method via Momentum contrast (MoCo) and temporal-consistent sampling. We leverage temporal-consistent sampling into MoCo to improve the ability of feature presentation on each input modality of compressed video. Modality-oriented fine-tuning is introduced to applying into the downstream compressed video action recognition. Extensive experiments demonstrate the effectiveness of our method on different datasets with different backbones. Compared to SOTA self-supervised learning methods for decoded videos on HMDB51 dataset, our method achieves the highest accuracy of 57.8\%.",

keywords = "Action recognition, Compressed video, Contrastive learning, Temporal-consistent sampling",

author = "Pan Chen and Shaohui Lin and Yongxiang Zhang and Jiachen Xu and Xin Tan and Lizhuang Ma",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 28th International Conference on Neural Information Processing, ICONIP 2021 ; Conference date: 08-12-2021 Through 12-12-2021",

year = "2021",

doi = "10.1007/978-3-030-92273-3\_20",

language = "英语",

isbn = "9783030922726",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "237--249",

editor = "Teddy Mantoro and Minho Lee and Ayu, \{Media Anugerah\} and Wong, \{Kok Wai\} and Hidayanto, \{Achmad Nizar\}",

booktitle = "Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings",

address = "德国",

}

Chen, P, Lin, S, Zhang, Y, Xu, J, Tan, X & Ma, L 2021, Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling. in T Mantoro, M Lee, MA Ayu, KW Wong & AN Hidayanto (eds), Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13111 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 237-249, 28th International Conference on Neural Information Processing, ICONIP 2021, Virtual, Online, 8/12/21. https://doi.org/10.1007/978-3-030-92273-3_20

Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling. / Chen, Pan; Lin, Shaohui; Zhang, Yongxiang et al.
Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings. ed. / Teddy Mantoro; Minho Lee; Media Anugerah Ayu; Kok Wai Wong; Achmad Nizar Hidayanto. Springer Science and Business Media Deutschland GmbH, 2021. p. 237-249 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13111 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling

AU - Chen, Pan

AU - Lin, Shaohui

AU - Zhang, Yongxiang

AU - Xu, Jiachen

AU - Tan, Xin

AU - Ma, Lizhuang

PY - 2021

Y1 - 2021

N2 - Compressed video action recognition targets at classifying action class in compressed video, instead of decoded/standard video. It benefits from fast training and inference by reducing the utilization of redundant information. However, off-the-shelf methods still rely on heavy-cost labels for training. In this paper, we propose self-supervised compressed video action recognition method via Momentum contrast (MoCo) and temporal-consistent sampling. We leverage temporal-consistent sampling into MoCo to improve the ability of feature presentation on each input modality of compressed video. Modality-oriented fine-tuning is introduced to applying into the downstream compressed video action recognition. Extensive experiments demonstrate the effectiveness of our method on different datasets with different backbones. Compared to SOTA self-supervised learning methods for decoded videos on HMDB51 dataset, our method achieves the highest accuracy of 57.8%.

AB - Compressed video action recognition targets at classifying action class in compressed video, instead of decoded/standard video. It benefits from fast training and inference by reducing the utilization of redundant information. However, off-the-shelf methods still rely on heavy-cost labels for training. In this paper, we propose self-supervised compressed video action recognition method via Momentum contrast (MoCo) and temporal-consistent sampling. We leverage temporal-consistent sampling into MoCo to improve the ability of feature presentation on each input modality of compressed video. Modality-oriented fine-tuning is introduced to applying into the downstream compressed video action recognition. Extensive experiments demonstrate the effectiveness of our method on different datasets with different backbones. Compared to SOTA self-supervised learning methods for decoded videos on HMDB51 dataset, our method achieves the highest accuracy of 57.8%.

KW - Action recognition

KW - Compressed video

KW - Contrastive learning

KW - Temporal-consistent sampling

UR - https://www.scopus.com/pages/publications/85121907592

U2 - 10.1007/978-3-030-92273-3_20

DO - 10.1007/978-3-030-92273-3_20

M3 - 会议稿件

AN - SCOPUS:85121907592

SN - 9783030922726

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 237

EP - 249

BT - Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings

A2 - Mantoro, Teddy

A2 - Lee, Minho

A2 - Ayu, Media Anugerah

A2 - Wong, Kok Wai

A2 - Hidayanto, Achmad Nizar

PB - Springer Science and Business Media Deutschland GmbH

T2 - 28th International Conference on Neural Information Processing, ICONIP 2021

Y2 - 8 December 2021 through 12 December 2021

ER -

Chen P, Lin S, Zhang Y, Xu J, Tan X, Ma L. Self-supervised Compressed Video Action Recognition via Temporal-Consistent Sampling. In Mantoro T, Lee M, Ayu MA, Wong KW, Hidayanto AN, editors, Neural Information Processing - 28th International Conference, ICONIP 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 237-249. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-92273-3_20