Customizing Visual-Language Foundation Models for Multi-modal Anomaly Detection and Reasoning
arxiv(2024)
摘要
Anomaly detection is vital in various industrial scenarios, including the
identification of unusual patterns in production lines and the detection of
manufacturing defects for quality control. Existing techniques tend to be
specialized in individual scenarios and lack generalization capacities. In this
study, we aim to develop a generic anomaly detection model applicable across
multiple scenarios. To achieve this, we customize generic visual-language
foundation models that possess extensive knowledge and robust reasoning
abilities into anomaly detectors and reasoners. Specifically, we introduce a
multi-modal prompting strategy that incorporates domain knowledge from experts
as conditions to guide the models. Our approach considers multi-modal prompt
types, including task descriptions, class context, normality rules, and
reference images. In addition, we unify the input representation of
multi-modality into a 2D image format, enabling multi-modal anomaly detection
and reasoning. Our preliminary studies demonstrate that combining visual and
language prompts as conditions for customizing the models enhances anomaly
detection performance. The customized models showcase the ability to detect
anomalies across different data modalities such as images and point clouds.
Qualitative case studies further highlight the anomaly detection and reasoning
capabilities, particularly for multi-object scenes and temporal data. Our code
is available at https://github.com/Xiaohao-Xu/Customizable-VLM.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要