TY - GEN
T1 - Domain Knowledge is All You Need
T2 - 46th International Conference on Software Engineering: Companion, ICSE-Companion 2024
AU - Xue, Zhiyi
AU - Li, Liangguo
AU - Tian, Senyue
AU - Chen, Xiaohong
AU - Chen, Liangyu
AU - Zhang, Min
AU - Li, Pingping
AU - Jiang, Tingting
N1 - Publisher Copyright:
© 2024 IEEE Computer Society. All rights reserved.
PY - 2024/5/23
Y1 - 2024/5/23
N2 - Despite the promise of automation, general-purpose Large Language Models (LLMs) face difficulties in generating complete and accurate test cases from informal software requirements, primarily due to challenges in interpreting unstructured text and producing diverse, relevant scenarios. This paper argues that incorporating domain knowledge significantly improves LLM performance in test case generation. We report on the successful deployment of our LLM-powered tool, LLM4Fin, in the FinTech domain, showcasing the crucial role of domain knowledge in addressing the aforementioned challenges.We demonstrate two methods for integrating domain knowledge: implicit incorporation through model fine-tuning, and explicit incorporation with algorithm design. This combined approach delivers remarkable results, achieving up to 98.18% improvement in test scenario coverage and reducing generation time from 20 minutes to 7 seconds.
AB - Despite the promise of automation, general-purpose Large Language Models (LLMs) face difficulties in generating complete and accurate test cases from informal software requirements, primarily due to challenges in interpreting unstructured text and producing diverse, relevant scenarios. This paper argues that incorporating domain knowledge significantly improves LLM performance in test case generation. We report on the successful deployment of our LLM-powered tool, LLM4Fin, in the FinTech domain, showcasing the crucial role of domain knowledge in addressing the aforementioned challenges.We demonstrate two methods for integrating domain knowledge: implicit incorporation through model fine-tuning, and explicit incorporation with algorithm design. This combined approach delivers remarkable results, achieving up to 98.18% improvement in test scenario coverage and reducing generation time from 20 minutes to 7 seconds.
UR - https://www.scopus.com/pages/publications/85194896135
U2 - 10.1145/3639478.3643087
DO - 10.1145/3639478.3643087
M3 - 会议稿件
AN - SCOPUS:85194896135
T3 - Proceedings - International Conference on Software Engineering
SP - 314
EP - 315
BT - Proceedings - 2024 ACM/IEEE 46th International Conference on Software Engineering
PB - IEEE Computer Society
Y2 - 14 April 2024 through 20 April 2024
ER -