Deep Learning Pipeline for Spotting Macro- and Micro-expressions in Long Video Sequences Based on Action Units and Optical Flow

•For the first time, AUs and optical flow features are combined to spot either macro- or micro- expression intervals.•The proposal can eliminate the influence of facial image change caused by noises, such as body or head movement.•The proposed Concat-CNN model can learn both the inner features of a single frame and the correlation between frames.•The re-labeling method considers the overall change process of a specific expression and improves the detection performance.•The proposal shows remarkable improvement in the F1 scores on datasets, such as the CAS(ME)2-cropped and the SAMM-LV.