Actor-Aware Contrastive Learning for Semi-Supervised Action Recognition

The unique features of existing video datasets have led self-supervised contrastive learning to scene correlations and background biases, resulting in poor generalization in scene-invariant action recognition. Therefore, we propose Actor-aware Contrastive Learning for semi-supervised action recognition (ActorCLR). We employ localized actors to encourage the model to learn discriminative regions and mitigate the model's reliance on the video background during contrastive training. Furthermore, we introduce Inter-video Background Mixing (iBM) augmentation strategy to inject scene-invariance into the model. For iBM, we patch inter-video crops of four randomly selected frames to create a distinct frame for each video individually. The patched frame is mixed with the target video frames to produce a spatially distorted sample. Then, we jointly optimize contrastive loss and consistency regularization with localized actors and corresponding iBM-augmented videos in a semi-supervised manner. iBM also mixes the one-hot-encoded labels of patches with the target video's label, which softens the decision boundaries of the semi-supervised model. Our experimental results show that ActorCLR significantly improved action recognition on Kinetics-400, UCF101, and HMDB51 datasets under a low-label regime.