Domain Adaptive Representation Learning for Facial Action Unit Recognition

•Facial action unit recognition system is improved by a novel approach that learns a coordinated representation through cross-modality supervision.•The proposed method shows superior generalization capabilities of the trained latent representation when applied to datasets lacking thermal images, complementing the performance of the dataset-specific action unit recognizers.•The transfer learning technique that was proposed in order to adapt the learned representation to the target domain improved performance on target domain datasets.•Multimodal conditional feature enhancement method, which is presented, proves that creating a unified representation for fusion, may not be required if both source modalities are present.