Summarizing Charts of Financial Document via Context-Aware Multi-Modeling

In the field of financial analysis, investment research analysts depend on a detailed understanding of complex financial documents to guide their decision-making process. Charts, while providing visual insights into data, present challenges in summarization. To address this issue, we present a novel approach that leverages contextual awareness, both in terms of textual semantics and visual perception. Our method begins with object detection technology to accurately locate and identify charts. Subsequently, a pre-trained language model is employed for vectorizing text and chart captions, enabling effective correlation between charts and their textual descriptions. Utilizing a large language model and strategic prompt engineering, we generate concise yet informative chart summaries, and incorporate visual saliency to assign scores, quantifying the importance of each chart for more effective data interpretation. Our study, supported by dedicated datasets, validates efficiency and accuracy improvements in financial analysis, expediting well-informed investment decisions.