Informative Prompt Learning for Low-Shot Commonsense Question Answering via Fine-Grained Redundancy Reduction

Low-shot commonsense question answering (CQA) poses a big challenge due to the absence of sufficient labeled data and commonsense knowledge. Recent work focuses on utilizing the potential of commonsense reasoning of pre-trained language models (PLMs) for low-shot CQA. In addition, various prompt learning methods have been studied to elicit implicit knowledge from PLMs for performance promotion. Whereas, it has been shown that PLMs suffer from the redundancy problem that many neurons encode similar information, especially under a small sample regime, leading prompt learning to be less informative in low-shot scenarios. In this paper, we propose an informative prompt learning approach, which aims to elicit more diverse and useful knowledge from PLMs for low-shot CQA via fine-grained redundancy reduction. Specifically, our redundancy-reduction method imposes restrictions upon the fine-grained neuron-level to encourage each dimension to model different knowledge or clues. Experiments on three benchmark datasets show the great advantages of our proposed approach in low-shot settings. Moreover, we conduct both quantitative and qualitative analyses, which shed light on why our approach can lead to great improvements.