Multi-Modal Adversarial Example Detection with Transformer

Although deep neural networks have shown great potential for many tasks, they are vulnerable to adversarial examples, which are generated by adding small perturbations to natural examples. Recently, many studies have proved that making full use of different modalities can effectively enhance the representational ability of deep neural networks. We propose a multi-modal deep fusion Transformer, termed MDFT. First, the audio feature and the rich semantic text features are extracted by audio encoders and text encoders, respectively. Then, multi-modal attention mechanisms are established to capture the high-level interactions between the audio and linguistic domains to obtain joint multi-modal representation. Finally, the representation is propagated to a dense layer to generate the detection result. The accuracy of this model compared with its unimodal variant on WiAd dataset and BlAd dataset are improved by 0.12 % and 0.19 %, respectively. Experimental results on the two datasets show that MDFT outperforms its unimodal variant model.