A semantically driven self-supervised algorithm for detecting anomalies in image sets

Humans can readily detect when an image does not belong in a set by comparing semantic information between images to derive meaning and relationships from the colors, shapes, sizes, locations, and textures within them. Current self-supervised anomaly detection algorithms in computer vision do not possess this ability. Most algorithms learn to detect anomalies through a training objective that only optimizes for machine-level features and do not evaluate semantic features. To fill this gap, we propose a novel self-supervised algorithm that detects anomalies by learning and modeling semantic information within a set of images. This is accomplished by first training our algorithm to be sensitive or invariant to targeted semantic information, and then modeling the semantic relationships learned so that we can detect anomalies by measuring how far images deviate from the model. Experimenting with our algorithm, we show that different datasets can have different semantic sensitivities, those sensitivities can fluctuate between and within sets depending on different aspects of the images, and directly targeting those sensitivities can improve anomaly detection performance. Using our algorithm, we were able to achieve a AUROC of 0.7146 on the CIFAR-10 dataset, which is an improvement of 0.0741 over another current leading self-supervised anomaly detection algorithm.