MeFormer: Generating Radiology Reports via Memory Enhanced Pretraining Transformer

Writing a radiology image report is a very time-consuming and tedious task. Using AI to generate the report is an efficient approach, but there are still two significant challenges. First, the model requires to be fine-tuned regularly with the increasing number of patients; Secondly, the quality of text generation needs to be improved because medical observations are complex. In order to solve above challenges, we propose Memory Enhanced Pretraining Transformer (MeFormer). It uses the pretrained Vision Transformer, which efficiently reduces the number of training parameters and transfers fruitful knowledge for the downstream task. At the same time, memory module was introduced into Transformer. The salient pattern in radiology reports are memorized through this design, and they can serve as cross-references during text generation, moderately enhancing the quality of generated diagnostic reports. Extensive experiments on two datasets show that our method achieves comparable performance to other state-of-the-art methods.