Fine-grained image classification with factorized deep user click feature

•We propose a novel image classification framework with deep click feature. Compared with previous visual-feature-based approaches, the proposed approach captures image semantics in a better manner.•We construct factorized TF-IDF vectors using word vocabularies of different POS to represent images. Compared with traditional TF-IDF vector built on an entire word vocabulary, the factorized TF-IDF vectors discover better semantics with complementary information of different POS.•We develop the click feature tensor to integrate the factorized TF-IDF vectors, and devise an end-to-end deep neural network on the click feature tensors to learn a structured click feature. The deep click model further enhances image representations by capturing hierarchical semantics.