Task-dependent contribution of higher-order statistics to natural texture processing

During natural visual behavior our visual system extracts multiple features from its inputs and uses them to solve different tasks. Each feature conveys relevant information to different tasks, and for a given task our visual system relies on the relevant features while ignoring others. However, this hypothesis remains largely untested for complex tasks with natural stimuli. Here we compare the role of spectral and higher order (HOS) texture statistics of the Portilla-Simoncelli model across tasks using natural images, to explain their task-dependent use by humans. Portilla-Simoncelli HOS are important for human texture perception, peripheral vision, and physiology, but they play a much smaller role for texture segmentation. Modeling work suggests this could reflect the redundancy between HOS and spectral statistics (a strong segmentation cue in humans) for natural image segmentation. But the importance of HOS for texture perception suggests that these statistics may be informative for other texture-related tasks. In this work we test the hypothesis that, in contrast to segmentation, HOS are superior to spectral statistics for natural texture classification. To test this, we trained linear classifiers to solve 4 different natural image classification tasks (classification of physical texture instances, materials, perceptual descriptions and scenes) across 11 datasets, and compared the performance afforded by HOS and spectral statistics. We find that HOS improved task performance considerably over spectral statistics, unlike what was reported for segmentation. This is compatible with an account of the task-dependent use of these features by humans based on their task-dependent relevance in natural images. Interestingly, we find that the contribution of HOS varies between classification tasks, with larger improvements for instance classification. Future work should test whether use of HOS by humans follows this finer pattern within classification, and explore the computational underpinnings of the varying HOS contributions.