How Does Self-supervised Learning Work? A Representation Learning Perspective

Self-supervised learning (SSL) is a popular machine learning paradigm that utilizes a large amount of unlabeled data to facilitate the learning from a small number of labeled data. While SSL has achieved great success in different tasks, its theoretical understanding remains largely open. In this paper, we aim to theoretically understand a special kind of SSL approaches based on pre-training and fine-tuning. In particular, the SSL approach we consider first trains a neural network based on the unlabeled data with help of pseudo labelers. Then it fine-tunes the pre-trained network on a small amount of labeled data. We prove that, under certain data and neural network models, SSL can achieve nearly zero test loss, while a neural network directly trained by supervised learning on the same amount of labeled data can only achieve constant test loss. Our theoretical result demonstrates a separation between SSL and supervised learning on the same amount of labeled data and sheds light on the essence of representation learning for the success of SSL.