A hybrid model with pre-trained entity-aware transformer for relation extraction

Distantly supervised relation extraction is an efficient method to extract novel relational facts from unstructed text. Most previous neural methods adopt Convolutional Neural Network (CNN) or Recurrent Neural Network (RNN) to encode sentences. However, CNN is difficult to learn long-range dependencies and the parallelization of training RNN is precluded by its sequential nature. In this paper, we propose a novel hybrid model that combines Piece-wise Convolutional Neural Network (PCNN) and Entity-Aware Transformer to extract local features and learn the dependencies between distant positions jointly. The entity-aware Transformer is able to take semantic and syntax information under consideration and acquire entity-specific representations. The inner-sentence attention mechanism is then used over Transformer to alleviate the noise caused by irrelevant words. We concatenate outputs of PCNN and Transformer with word embeddings of entity mentions and then send them to the classifier, which can boost the performance of our model further. A transfer learning based strategy is applied, where the entity-aware Transformer is initialized with a priori knowledge learned from the related task of entity typing to improve the robustness of our model. The experimental results on a large-scale benchmark dataset show that our hybrid model with the pre-training strategy gets AUC score of 0.432 and outperforms the state-of-the-art baselines.