Democratizing Fine-grained Visual Recognition with Large Language Models
CoRR(2024)
摘要
Identifying subordinate-level categories from images is a longstanding task
in computer vision and is referred to as fine-grained visual recognition
(FGVR). It has tremendous significance in real-world applications since an
average layperson does not excel at differentiating species of birds or
mushrooms due to subtle differences among the species. A major bottleneck in
developing FGVR systems is caused by the need of high-quality paired expert
annotations. To circumvent the need of expert knowledge we propose Fine-grained
Semantic Category Reasoning (FineR) that internally leverages the world
knowledge of large language models (LLMs) as a proxy in order to reason about
fine-grained category names. In detail, to bridge the modality gap between
images and LLM, we extract part-level visual attributes from images as text and
feed that information to a LLM. Based on the visual attributes and its internal
world knowledge the LLM reasons about the subordinate-level category names. Our
training-free FineR outperforms several state-of-the-art FGVR and language and
vision assistant models and shows promise in working in the wild and in new
domains where gathering expert annotation is arduous.
更多查看译文
关键词
Vision-Language Models,Large Language Models,Prompting,Multimodal,Fine-grained Visual Recognition
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要