Constrained and directional ensemble attention for facial action unit detection

Facial action unit (AU) detection is a challenging task, due to the subtlety of each AU in local area and the correlations among AUs in global face. In recent years, the prevailing attention mechanism has been introduced to AU detection. However, the inherent mechanism of self-attention weight distribution has been rarely explored. Besides, ensemble learning is an efficient technique, but gains little attention in AU detection. Considering the above limitations, we propose a local self-attention constraining (LSC) network, by regarding the self-attention distribution of each AU as a spatial distribution, and constraining it based on prior knowledge so as to capture AU-related local information. Moreover, to learn correlations among different AU regions, we propose a global dual-directional attention (GDA) network, which adaptively learns global attention map from both vertical and horizontal directions. Last but not least, the two networks from different views of capturing patterns are assembled to integrate both advantages. Extensive experiments on BP4D, DISFA, and GFT benchmarks demonstrate that our methods including local self-attention constraining, global dual-directional attention, and multi-view ensemble all significantly surpass state-of-the-art AU detection works.