Revealing the locus and content of behaviorally relevant information about real-world scenes in human visual cortex

Scene information can be rapidly categorized and translated into appropriate actions. While there has been substantial progress in understanding how scene information is represented in the brain, it remains unknown to what extent particular scene representations are relevant for decision behavior. To address this question, we recorded fMRI data while human participants (N=29) viewed manmade and natural scenes and paired it with behavioral data recorded in a separate session from participants (N=30) performing either a categorization task or an orthogonal task on the same stimuli. In order to identify behaviorally relevant information, we correlated the reaction times (RTs) of individual scenes with the distances of scene-specific fMRI responses to a hyperplane derived from a multivariate pattern classifier. Our findings are threefold. First, we found negative distance-RT correlations for the categorization task in early visual cortex, lateral occipital complex but not in the parahippocampal place area, in line with a representation of behaviorally relevant information. Second, correlating the neural distances with the RTs from the orthogonal task resulted in positive distance-RT correlations in early visual cortex and lateral occipital complex. In contrast to the results from the categorization task, this positive relationship indicates interference between scene representations and the orthogonal task. Finally, to characterize the content of behaviorally relevant information, we assessed the shared variance between neural distances, categorization RTs and distances derived from several artificial neural network architectures. Across networks, we found the highest shared variance for low to intermediate network layers, suggesting that low to mid-level visual features account best for the behaviorally relevant information. By linking neural representations and visual features derived from computational models of scene recognition to decision behavior, these results reveal the spatial extent and content of behaviorally relevant information about real-world scenes in visual cortex.