INFER: Distilling knowledge from human-generated rules with uncertainty for STINs

As a long-time wish, researchers always want to find a way to fuse human knowledge directly into machine models that can fulfill intelligent tasks. Existing researches attempted to approach this goal by manually labeling data or writing simple rules without considering the human's inherent ability to estimate uncertainty. These approaches cannot take full advantage of human abilities and hence make the overall system inefficient. In this paper, we propose a novel approach INFER that can distill knowledge from humans in the form of rules with uncertainty. Firstly, we propose a new paradigm of providing human intelligence by expressing human knowledge as uncertain rules. After obtaining the set of rules, we design a rule aggregation model and a classification model, these two models are jointly trained in a knowledge distillation framework with the uncertainty knowledge. We further improve the distillation process with a curriculum learning-based training method. Through these, we can directly extract knowledge from inaccurate human knowledge. Empirical results on four different tasks demonstrate our proposed INFER approach can significantly improve model performance. Furthermore, the proposed uncertainty can provide more information and be effective in refining rules.