Element Information Enhancement for Diagram Question Answering with Synthetic Data

Yadong Zhang; Yang Chen; Yupei Ren; Man Lan; Yuefeng Chen

doi:10.1007/978-981-19-8300-9_9

Element Information Enhancement for Diagram Question Answering with Synthetic Data

Yadong Zhang
, Yang Chen
, Yupei Ren
, Man Lan^*
, Yuefeng Chen

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Unlike natural pictures, diagrams are a highly abstract vehicle for knowledge representation, and Diagram Question Answering involves complex reasoning processes such as diagram element detection. However, due to low resource constraints, achieving efficient extraction of diagram elements is challenging. In addition, vision tasks rely on image feature extraction, and most feature extraction today is based on real scenario images on ImageNet. To solve the above problems, we programmatically synthesized a diagram dataset to implement diagram element prediction and put its feature extraction module to use on downstream task. In the actual task, we explicitly input the predicted image elements from the diagram into the model. The experimental comparison shows a significant improvement in our model compared to the baseline.

Original language	English
Title of host publication	CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers
Editors	Ningyu Zhang, Meng Wang, Tianxing Wu, Wei Hu, Shumin Deng
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	78-86
Number of pages	9
ISBN (Print)	9789811982996
DOIs	https://doi.org/10.1007/978-981-19-8300-9_9
State	Published - 2022
Event	7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022 - Qinhuangdao, China Duration: 24 Aug 2022 → 27 Aug 2022

Publication series

Name	Communications in Computer and Information Science
Volume	1711 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022
Country/Territory	China
City	Qinhuangdao
Period	24/08/22 → 27/08/22

Keywords

Data synthesis
Diagram question answering
Low resource

Access to Document

10.1007/978-981-19-8300-9_9

Cite this

Zhang, Y., Chen, Y., Ren, Y., Lan, M., & Chen, Y. (2022). Element Information Enhancement for Diagram Question Answering with Synthetic Data. In N. Zhang, M. Wang, T. Wu, W. Hu, & S. Deng (Eds.), CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers (pp. 78-86). (Communications in Computer and Information Science; Vol. 1711 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-19-8300-9_9

Zhang, Yadong ; Chen, Yang ; Ren, Yupei et al. / Element Information Enhancement for Diagram Question Answering with Synthetic Data. CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers. editor / Ningyu Zhang ; Meng Wang ; Tianxing Wu ; Wei Hu ; Shumin Deng. Springer Science and Business Media Deutschland GmbH, 2022. pp. 78-86 (Communications in Computer and Information Science).

@inproceedings{939b0209ee6640b6b114433cf495d71c,

title = "Element Information Enhancement for Diagram Question Answering with Synthetic Data",

abstract = "Unlike natural pictures, diagrams are a highly abstract vehicle for knowledge representation, and Diagram Question Answering involves complex reasoning processes such as diagram element detection. However, due to low resource constraints, achieving efficient extraction of diagram elements is challenging. In addition, vision tasks rely on image feature extraction, and most feature extraction today is based on real scenario images on ImageNet. To solve the above problems, we programmatically synthesized a diagram dataset to implement diagram element prediction and put its feature extraction module to use on downstream task. In the actual task, we explicitly input the predicted image elements from the diagram into the model. The experimental comparison shows a significant improvement in our model compared to the baseline.",

keywords = "Data synthesis, Diagram question answering, Low resource",

author = "Yadong Zhang and Yang Chen and Yupei Ren and Man Lan and Yuefeng Chen",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.; 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022 ; Conference date: 24-08-2022 Through 27-08-2022",

year = "2022",

doi = "10.1007/978-981-19-8300-9\_9",

language = "英语",

isbn = "9789811982996",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "78--86",

editor = "Ningyu Zhang and Meng Wang and Tianxing Wu and Wei Hu and Shumin Deng",

booktitle = "CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers",

address = "德国",

}

Zhang, Y, Chen, Y, Ren, Y, Lan, M & Chen, Y 2022, Element Information Enhancement for Diagram Question Answering with Synthetic Data. in N Zhang, M Wang, T Wu, W Hu & S Deng (eds), CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers. Communications in Computer and Information Science, vol. 1711 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 78-86, 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Qinhuangdao, China, 24/08/22. https://doi.org/10.1007/978-981-19-8300-9_9

Element Information Enhancement for Diagram Question Answering with Synthetic Data. / Zhang, Yadong; Chen, Yang; Ren, Yupei et al.
CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers. ed. / Ningyu Zhang; Meng Wang; Tianxing Wu; Wei Hu; Shumin Deng. Springer Science and Business Media Deutschland GmbH, 2022. p. 78-86 (Communications in Computer and Information Science; Vol. 1711 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Element Information Enhancement for Diagram Question Answering with Synthetic Data

AU - Zhang, Yadong

AU - Chen, Yang

AU - Ren, Yupei

AU - Lan, Man

AU - Chen, Yuefeng

PY - 2022

Y1 - 2022

N2 - Unlike natural pictures, diagrams are a highly abstract vehicle for knowledge representation, and Diagram Question Answering involves complex reasoning processes such as diagram element detection. However, due to low resource constraints, achieving efficient extraction of diagram elements is challenging. In addition, vision tasks rely on image feature extraction, and most feature extraction today is based on real scenario images on ImageNet. To solve the above problems, we programmatically synthesized a diagram dataset to implement diagram element prediction and put its feature extraction module to use on downstream task. In the actual task, we explicitly input the predicted image elements from the diagram into the model. The experimental comparison shows a significant improvement in our model compared to the baseline.

AB - Unlike natural pictures, diagrams are a highly abstract vehicle for knowledge representation, and Diagram Question Answering involves complex reasoning processes such as diagram element detection. However, due to low resource constraints, achieving efficient extraction of diagram elements is challenging. In addition, vision tasks rely on image feature extraction, and most feature extraction today is based on real scenario images on ImageNet. To solve the above problems, we programmatically synthesized a diagram dataset to implement diagram element prediction and put its feature extraction module to use on downstream task. In the actual task, we explicitly input the predicted image elements from the diagram into the model. The experimental comparison shows a significant improvement in our model compared to the baseline.

KW - Data synthesis

KW - Diagram question answering

KW - Low resource

UR - https://www.scopus.com/pages/publications/85144225481

U2 - 10.1007/978-981-19-8300-9_9

DO - 10.1007/978-981-19-8300-9_9

M3 - 会议稿件

AN - SCOPUS:85144225481

SN - 9789811982996

T3 - Communications in Computer and Information Science

SP - 78

EP - 86

BT - CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers

A2 - Zhang, Ningyu

A2 - Wang, Meng

A2 - Wu, Tianxing

A2 - Hu, Wei

A2 - Deng, Shumin

PB - Springer Science and Business Media Deutschland GmbH

T2 - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022

Y2 - 24 August 2022 through 27 August 2022

ER -

Zhang Y, Chen Y, Ren Y, Lan M, Chen Y. Element Information Enhancement for Diagram Question Answering with Synthetic Data. In Zhang N, Wang M, Wu T, Hu W, Deng S, editors, CCKS 2022 - Evaluation Track - 7th China Conference on Knowledge Graph and Semantic Computing Evaluations, CCKS 2022, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 78-86. (Communications in Computer and Information Science). doi: 10.1007/978-981-19-8300-9_9