Overview of the PromptCBLUE Shared Task in CHIP2023

Wei Zhu; Xiaoling Wang; Mosha Chen; Buzhou Tang

doi:10.1007/978-981-97-1717-0_1

Overview of the PromptCBLUE Shared Task in CHIP2023

Wei Zhu^*, Xiaoling Wang, Mosha Chen, Buzhou Tang

^*Corresponding author for this work

School of Computer Science and Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

This paper presents an overview of the PromptCBLUE shared task (http://cips-chip.org.cn/2023/eval1) held in the CHIP-2023 Conference. This shared task reformulates the CBLUE benchmark, and provide a good testbed for Chinese open-domain or medical-domain large language models (LLMs) in general medical natural language processing. Two different tracks are held: (a) prompt tuning track, investigating the multitask prompt tuning of LLMs, (b) probing the in-context learning capabilities of open-sourced LLMs. Many teams from both the industry and academia participated in the shared tasks, and the top teams achieved amazing test results. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks. Finally, the paper summarizes the techniques and results of the evaluation of the various approaches explored by the participating teams.

Original language	English
Title of host publication	Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings
Editors	Hua Xu, Qingcai Chen, Hongfei Lin, Fei Wu, Lei Liu, Buzhou Tang, Tianyong Hao, Zhengxing Huang, Jianbo Lei, Zuofeng Li, Hui Zong
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	3-20
Number of pages	18
ISBN (Print)	9789819717163
DOIs	https://doi.org/10.1007/978-981-97-1717-0_1
State	Published - 2024
Event	Evaluation track of the 9th China Health Information Processing Conference, CHIP 2023 - Hangzhou, China Duration: 27 Oct 2023 → 29 Oct 2023

Publication series

Name	Communications in Computer and Information Science
Volume	2080 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	Evaluation track of the 9th China Health Information Processing Conference, CHIP 2023
Country/Territory	China
City	Hangzhou
Period	27/10/23 → 29/10/23

Keywords

Large language models
Medical natural language processing
PromptCBLUE
in-context learning
parameter efficient fine-tuning

Access to Document

10.1007/978-981-97-1717-0_1

Cite this

Zhu, W., Wang, X., Chen, M., & Tang, B. (2024). Overview of the PromptCBLUE Shared Task in CHIP2023. In H. Xu, Q. Chen, H. Lin, F. Wu, L. Liu, B. Tang, T. Hao, Z. Huang, J. Lei, Z. Li, & H. Zong (Eds.), Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings (pp. 3-20). (Communications in Computer and Information Science; Vol. 2080 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-97-1717-0_1

Zhu, Wei ; Wang, Xiaoling ; Chen, Mosha et al. / Overview of the PromptCBLUE Shared Task in CHIP2023. Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings. editor / Hua Xu ; Qingcai Chen ; Hongfei Lin ; Fei Wu ; Lei Liu ; Buzhou Tang ; Tianyong Hao ; Zhengxing Huang ; Jianbo Lei ; Zuofeng Li ; Hui Zong. Springer Science and Business Media Deutschland GmbH, 2024. pp. 3-20 (Communications in Computer and Information Science).

@inproceedings{063a0b52559c480e9c4dfe58be38aac8,

title = "Overview of the PromptCBLUE Shared Task in CHIP2023",

abstract = "This paper presents an overview of the PromptCBLUE shared task (http://cips-chip.org.cn/2023/eval1) held in the CHIP-2023 Conference. This shared task reformulates the CBLUE benchmark, and provide a good testbed for Chinese open-domain or medical-domain large language models (LLMs) in general medical natural language processing. Two different tracks are held: (a) prompt tuning track, investigating the multitask prompt tuning of LLMs, (b) probing the in-context learning capabilities of open-sourced LLMs. Many teams from both the industry and academia participated in the shared tasks, and the top teams achieved amazing test results. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks. Finally, the paper summarizes the techniques and results of the evaluation of the various approaches explored by the participating teams.",

keywords = "Large language models, Medical natural language processing, PromptCBLUE, in-context learning, parameter efficient fine-tuning",

author = "Wei Zhu and Xiaoling Wang and Mosha Chen and Buzhou Tang",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024.; Evaluation track of the 9th China Health Information Processing Conference, CHIP 2023 ; Conference date: 27-10-2023 Through 29-10-2023",

year = "2024",

doi = "10.1007/978-981-97-1717-0\_1",

language = "英语",

isbn = "9789819717163",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "3--20",

editor = "Hua Xu and Qingcai Chen and Hongfei Lin and Fei Wu and Lei Liu and Buzhou Tang and Tianyong Hao and Zhengxing Huang and Jianbo Lei and Zuofeng Li and Hui Zong",

booktitle = "Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings",

address = "德国",

}

Zhu, W, Wang, X, Chen, M & Tang, B 2024, Overview of the PromptCBLUE Shared Task in CHIP2023. in H Xu, Q Chen, H Lin, F Wu, L Liu, B Tang, T Hao, Z Huang, J Lei, Z Li & H Zong (eds), Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings. Communications in Computer and Information Science, vol. 2080 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 3-20, Evaluation track of the 9th China Health Information Processing Conference, CHIP 2023, Hangzhou, China, 27/10/23. https://doi.org/10.1007/978-981-97-1717-0_1

Overview of the PromptCBLUE Shared Task in CHIP2023. / Zhu, Wei; Wang, Xiaoling; Chen, Mosha et al.
Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings. ed. / Hua Xu; Qingcai Chen; Hongfei Lin; Fei Wu; Lei Liu; Buzhou Tang; Tianyong Hao; Zhengxing Huang; Jianbo Lei; Zuofeng Li; Hui Zong. Springer Science and Business Media Deutschland GmbH, 2024. p. 3-20 (Communications in Computer and Information Science; Vol. 2080 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Overview of the PromptCBLUE Shared Task in CHIP2023

AU - Zhu, Wei

AU - Wang, Xiaoling

AU - Chen, Mosha

AU - Tang, Buzhou

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024.

PY - 2024

Y1 - 2024

N2 - This paper presents an overview of the PromptCBLUE shared task (http://cips-chip.org.cn/2023/eval1) held in the CHIP-2023 Conference. This shared task reformulates the CBLUE benchmark, and provide a good testbed for Chinese open-domain or medical-domain large language models (LLMs) in general medical natural language processing. Two different tracks are held: (a) prompt tuning track, investigating the multitask prompt tuning of LLMs, (b) probing the in-context learning capabilities of open-sourced LLMs. Many teams from both the industry and academia participated in the shared tasks, and the top teams achieved amazing test results. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks. Finally, the paper summarizes the techniques and results of the evaluation of the various approaches explored by the participating teams.

AB - This paper presents an overview of the PromptCBLUE shared task (http://cips-chip.org.cn/2023/eval1) held in the CHIP-2023 Conference. This shared task reformulates the CBLUE benchmark, and provide a good testbed for Chinese open-domain or medical-domain large language models (LLMs) in general medical natural language processing. Two different tracks are held: (a) prompt tuning track, investigating the multitask prompt tuning of LLMs, (b) probing the in-context learning capabilities of open-sourced LLMs. Many teams from both the industry and academia participated in the shared tasks, and the top teams achieved amazing test results. This paper describes the tasks, the datasets, evaluation metrics, and the top systems for both tasks. Finally, the paper summarizes the techniques and results of the evaluation of the various approaches explored by the participating teams.

KW - Large language models

KW - Medical natural language processing

KW - PromptCBLUE

KW - in-context learning

KW - parameter efficient fine-tuning

UR - https://www.scopus.com/pages/publications/85189635915

U2 - 10.1007/978-981-97-1717-0_1

DO - 10.1007/978-981-97-1717-0_1

M3 - 会议稿件

AN - SCOPUS:85189635915

SN - 9789819717163

T3 - Communications in Computer and Information Science

SP - 3

EP - 20

BT - Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings

A2 - Xu, Hua

A2 - Chen, Qingcai

A2 - Lin, Hongfei

A2 - Wu, Fei

A2 - Liu, Lei

A2 - Tang, Buzhou

A2 - Hao, Tianyong

A2 - Huang, Zhengxing

A2 - Lei, Jianbo

A2 - Li, Zuofeng

A2 - Zong, Hui

PB - Springer Science and Business Media Deutschland GmbH

T2 - Evaluation track of the 9th China Health Information Processing Conference, CHIP 2023

Y2 - 27 October 2023 through 29 October 2023

ER -

Zhu W, Wang X, Chen M, Tang B. Overview of the PromptCBLUE Shared Task in CHIP2023. In Xu H, Chen Q, Lin H, Wu F, Liu L, Tang B, Hao T, Huang Z, Lei J, Li Z, Zong H, editors, Health Information Processing. Evaluation Track Papers - 9th China Conference, CHIP 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 3-20. (Communications in Computer and Information Science). doi: 10.1007/978-981-97-1717-0_1