Toward Evaluating the Reliability of Deep-Neural-Network-Based IoT Devices

Nowadays, the impressive performance of deep neural networks (DNNs) greatly advances the development of Internet of Things (IoT) in diverse scenarios. However, the exceptional vulnerability of DNNs to adversarial attack leads IoT devices to be exposed to potential security issues. Up to now, since adversarial training empirically remains robust against gradient-based adversarial attacks, it is believed to be the most effective defense method. In this article, we find that adversarial examples generated by gradient-based adversarial attacks tend to be less imperceptible induced by the gradient-based optimization methods (adopted in the attacks) being difficult on searching the most effective adversarial examples (i.e., the global extreme points), which may lead to an inaccurate estimation for the effectiveness of the adversarial training. To overcome the inherent defect of gradient-based adversarial attacks, we propose a novel adversarial attack named nongradient attack (NGA), of which search strategy is effective but no longer depends on gradients to enhance the threat of adversarial examples. In detail, NGA first initializes the adversarial examples outside, rather than inside, of decision boundary to make them misclassified by the model and then, under without violation of misclassified condition, adjusts the adversarial examples toward the crafted direction to close the original examples. Extensive experiments show that NGA significantly outperforms the state-of-the-art adversarial attacks on attack success rate (ASR) by 2%-7%. Moreover, we propose a new evaluation metric, i.e., composite criterion (CC) based on both ASR and accuracy, to better measure the effectiveness of adversarial training. In the experiments, CC has shown to be a more comprehensive yet appropriate evaluation metric.