Deep Spatial and Channel Sliding Attention Patches for Pose-invariant Facial Expression Recognition

Abstract Pose-invariant facial expression recognition (FER) is an import yet challenging research topics in computer vision, especially with the introduction of pose change and self-occlusion, which makes the recognition results changing from one observational angle to another. In this paper, a newly deep sliding patch combined with spatial-channel attention network is designed for pose-invariant FER. The new network architecture consists of three main components, namely, the sliding patch (SP) model, deep channel squeeze-and-extraction model (SE) and spatial sliding patch attention (SPA) model. The sliding patch (SP) model is devised to find the best patch size and stride for pose-invariant FER. The sliding patch attention (SPA) model can guide the network pay more attention to the current local patches and adaptively estimate the significance of each local sliding patch. The squeeze-and-extraction (SE) operation was implemented prior to SPA model whose aims to shrink the number of feature maps and provide more salient semantic feature maps for SPA model. Extensive experiments were implemented on four pose robust expression datasets, namely, BU3DFEP1, BU3DFEP2, Multi-PIE, Pose-RAFDB, Pose-Affect, and the experimental results indicated the validity and feasibility of SPA-SE network.