When and how convolutional neural networks generalize to out-of-distribution category-viewpoint combinations

Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari,Tomotake Sasaki,Fredo Durand大牛学者,Hanspeter Pfister大牛学者,Xavier Boix


引用 7|浏览21
Object recognition and viewpoint estimation lie at the heart of visual understanding. Recent studies have suggested that convolutional neural networks (CNNs) fail to generalize to out-of-distribution (OOD) category-viewpoint combinations, that is, combinations not seen during training. Here we investigate when and how such OOD generalization may be possible by evaluating CNNs trained to classify both object category and three-dimensional viewpoint on OOD combinations, and identifying the neural mechanisms that facilitate such OOD generalization. We show that increasing the number of in-distribution combinations (data diversity) substantially improves generalization to OOD combinations, even with the same amount of training data. We compare learning category and viewpoint in separate and shared network architectures, and observe starkly different trends on in-distribution and OOD combinations, that is, while shared networks are helpful in distribution, separate networks significantly outperform shared ones at OOD combinations. Finally, we demonstrate that such OOD generalization is facilitated by the neural mechanism of specialization, that is, the emergence of two types of neuron-neurons selective to category and invariant to viewpoint, and vice versa. The combination of object recognition and viewpoint estimation is essential for visual understanding. However, convolutional neural networks often fail to generalize to object category-viewpoint combinations that were not seen during training. The authors investigate the impact of data diversity and architectural choices on the capability of generalizing to out-of-distribution combinations.
AI 理解论文