Resilient Abstractive Summarization Model with Adaptively Weighted Training Loss

Recently, abstractive summarization models are preferred over extractive summarization models as they can generate words that do not exist in the original text, whose summary descriptions are more flexible and natural. Neural network-based models learn the pattern of summary generation from the training data by modeling the relationship between the original text and the reference summary, which is very dependent on the reference summary. Although we intuitively feel that summary with higher Abstraction Degree (quantified by the number of words in the summary which do not appear in the original text) will be more general, manually generated summaries with high Abstraction Degree are most likely subliminally written with additional knowledge. It's difficult to learn the generation pattern of such reference summary using limited training data. What's more, such reference summaries can even harm the model performance. To this end, we design a learning method that can adaptively weighted difference training samples based on their Abstraction Degree, so that the model will pay less attention to the samples with higher Abstraction Degree. Experiments of LCSTS and CNN-DM dataset show that our method greatly improves the performance of the summarization model and is resilient in the face of training data containing low quality reference summaries.