Micro-expression recognition with layered relations and more input frames

Micro-expressions (MEs) which are types of spontaneous facial movements, are difficult to recognize due to their short duration and low intensity. Recent ME recognition methods typically depend on spatio-temporal features around the eyebrow and mouth regions where MEs occur. And these features are often extracted from the onset and apex frames in which the intensities of MEs are considered to be zero and high respectively. In this paper, we improve the effectiveness of the spatio-temporal features by proposing a two-layer encoder of Transformer to model the features' relations. In addition, the novel recognition scheme captures the more detailed motion dynamics of MEs by employing more frames rather than the onset and apex frames. The recognition scheme is further refined by developing a graph convolution network with a trainable adjacency matrix for Action Units (AUs). Extensive experiments on multiple public datasets demonstrate that our method has better or comparable performance to SOTA methods on multiple evaluation metrics.