A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction

Bo Xu; Shizhou Huang; Ming Du; Hongya Wang; Hui Song; Yanghua Xiao; Xin Lin

doi:10.1007/978-3-031-30675-4_40

A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction

Bo Xu, Shizhou Huang, Ming Du, Hongya Wang, Hui Song, Yanghua Xiao, Xin Lin

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

Recently, multimodal information extraction has gained increasing attention in social media understanding, as it helps to accomplish the task of information extraction by adding images as auxiliary information to solve the ambiguity problem caused by insufficient semantic information in short texts. Despite their success, current methods do not take full advantage of the information provided by the diverse representations of images. To address this problem, we propose a novel unified visual prompt tuning framework with Mixture-of-Experts to fuse different types of image representations for multimodal information extraction. Extensive experiments conducted on two different multimodal information extraction tasks demonstrate the effectiveness of our method. The source code can be found at https://github.com/xubodhu/VisualPT-MoE.

Original language	English
Title of host publication	Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings
Editors	Xin Wang, Maria Luisa Sapino, Wook-Shin Han, Amr El Abbadi, Gill Dobbie, Zhiyong Feng, Yingxiao Shao, Hongzhi Yin
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	544-554
Number of pages	11
ISBN (Print)	9783031306747
DOIs	https://doi.org/10.1007/978-3-031-30675-4_40
State	Published - 2023
Event	28th International Conference on Database Systems for Advanced Applications, DASFAA 2023 - Tianjin, China Duration: 17 Apr 2023 → 20 Apr 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13945 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	28th International Conference on Database Systems for Advanced Applications, DASFAA 2023
Country/Territory	China
City	Tianjin
Period	17/04/23 → 20/04/23

Keywords

Mixture-of-Experts
Multimodal information extraction
Prompt learning
Social media

Access to Document

10.1007/978-3-031-30675-4_40

Cite this

Xu, B., Huang, S., Du, M., Wang, H., Song, H., Xiao, Y., & Lin, X. (2023). A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction. In X. Wang, M. L. Sapino, W.-S. Han, A. El Abbadi, G. Dobbie, Z. Feng, Y. Shao, & H. Yin (Eds.), Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings (pp. 544-554). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13945 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-30675-4_40

Xu, Bo ; Huang, Shizhou ; Du, Ming et al. / A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction. Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. editor / Xin Wang ; Maria Luisa Sapino ; Wook-Shin Han ; Amr El Abbadi ; Gill Dobbie ; Zhiyong Feng ; Yingxiao Shao ; Hongzhi Yin. Springer Science and Business Media Deutschland GmbH, 2023. pp. 544-554 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{b1e2ec12e6794c0b8afc4ed7f954f938,

title = "A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction",

abstract = "Recently, multimodal information extraction has gained increasing attention in social media understanding, as it helps to accomplish the task of information extraction by adding images as auxiliary information to solve the ambiguity problem caused by insufficient semantic information in short texts. Despite their success, current methods do not take full advantage of the information provided by the diverse representations of images. To address this problem, we propose a novel unified visual prompt tuning framework with Mixture-of-Experts to fuse different types of image representations for multimodal information extraction. Extensive experiments conducted on two different multimodal information extraction tasks demonstrate the effectiveness of our method. The source code can be found at https://github.com/xubodhu/VisualPT-MoE.",

keywords = "Mixture-of-Experts, Multimodal information extraction, Prompt learning, Social media",

author = "Bo Xu and Shizhou Huang and Ming Du and Hongya Wang and Hui Song and Yanghua Xiao and Xin Lin",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 28th International Conference on Database Systems for Advanced Applications, DASFAA 2023 ; Conference date: 17-04-2023 Through 20-04-2023",

year = "2023",

doi = "10.1007/978-3-031-30675-4\_40",

language = "英语",

isbn = "9783031306747",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "544--554",

editor = "Xin Wang and Sapino, \{Maria Luisa\} and Wook-Shin Han and \{El Abbadi\}, Amr and Gill Dobbie and Zhiyong Feng and Yingxiao Shao and Hongzhi Yin",

booktitle = "Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings",

address = "德国",

}

Xu, B, Huang, S, Du, M, Wang, H, Song, H, Xiao, Y & Lin, X 2023, A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction. in X Wang, ML Sapino, W-S Han, A El Abbadi, G Dobbie, Z Feng, Y Shao & H Yin (eds), Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13945 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 544-554, 28th International Conference on Database Systems for Advanced Applications, DASFAA 2023, Tianjin, China, 17/04/23. https://doi.org/10.1007/978-3-031-30675-4_40

A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction. / Xu, Bo; Huang, Shizhou; Du, Ming et al.
Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. ed. / Xin Wang; Maria Luisa Sapino; Wook-Shin Han; Amr El Abbadi; Gill Dobbie; Zhiyong Feng; Yingxiao Shao; Hongzhi Yin. Springer Science and Business Media Deutschland GmbH, 2023. p. 544-554 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13945 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction

AU - Xu, Bo

AU - Huang, Shizhou

AU - Du, Ming

AU - Wang, Hongya

AU - Song, Hui

AU - Xiao, Yanghua

AU - Lin, Xin

PY - 2023

Y1 - 2023

N2 - Recently, multimodal information extraction has gained increasing attention in social media understanding, as it helps to accomplish the task of information extraction by adding images as auxiliary information to solve the ambiguity problem caused by insufficient semantic information in short texts. Despite their success, current methods do not take full advantage of the information provided by the diverse representations of images. To address this problem, we propose a novel unified visual prompt tuning framework with Mixture-of-Experts to fuse different types of image representations for multimodal information extraction. Extensive experiments conducted on two different multimodal information extraction tasks demonstrate the effectiveness of our method. The source code can be found at https://github.com/xubodhu/VisualPT-MoE.

AB - Recently, multimodal information extraction has gained increasing attention in social media understanding, as it helps to accomplish the task of information extraction by adding images as auxiliary information to solve the ambiguity problem caused by insufficient semantic information in short texts. Despite their success, current methods do not take full advantage of the information provided by the diverse representations of images. To address this problem, we propose a novel unified visual prompt tuning framework with Mixture-of-Experts to fuse different types of image representations for multimodal information extraction. Extensive experiments conducted on two different multimodal information extraction tasks demonstrate the effectiveness of our method. The source code can be found at https://github.com/xubodhu/VisualPT-MoE.

KW - Mixture-of-Experts

KW - Multimodal information extraction

KW - Prompt learning

KW - Social media

UR - https://www.scopus.com/pages/publications/85161647866

U2 - 10.1007/978-3-031-30675-4_40

DO - 10.1007/978-3-031-30675-4_40

M3 - 会议稿件

AN - SCOPUS:85161647866

SN - 9783031306747

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 544

EP - 554

BT - Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings

A2 - Wang, Xin

A2 - Sapino, Maria Luisa

A2 - Han, Wook-Shin

A2 - El Abbadi, Amr

A2 - Dobbie, Gill

A2 - Feng, Zhiyong

A2 - Shao, Yingxiao

A2 - Yin, Hongzhi

PB - Springer Science and Business Media Deutschland GmbH

T2 - 28th International Conference on Database Systems for Advanced Applications, DASFAA 2023

Y2 - 17 April 2023 through 20 April 2023

ER -

Xu B, Huang S, Du M, Wang H, Song H, Xiao Y et al. A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction. In Wang X, Sapino ML, Han WS, El Abbadi A, Dobbie G, Feng Z, Shao Y, Yin H, editors, Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 544-554. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-30675-4_40