Distilling Semantic Priors from SAM to Efficient Image Restoration Models
CVPR 2024(2024)
摘要
In image restoration (IR), leveraging semantic priors from segmentation
models has been a common approach to improve performance. The recent segment
anything model (SAM) has emerged as a powerful tool for extracting advanced
semantic priors to enhance IR tasks. However, the computational cost of SAM is
prohibitive for IR, compared to existing smaller IR models. The incorporation
of SAM for extracting semantic priors considerably hampers the model inference
efficiency. To address this issue, we propose a general framework to distill
SAM's semantic knowledge to boost exiting IR models without interfering with
their inference process. Specifically, our proposed framework consists of the
semantic priors fusion (SPF) scheme and the semantic priors distillation (SPD)
scheme. SPF fuses two kinds of information between the restored image predicted
by the original IR model and the semantic mask predicted by SAM for the refined
restored image. SPD leverages a self-distillation manner to distill the fused
semantic priors to boost the performance of original IR models. Additionally,
we design a semantic-guided relation (SGR) module for SPD, which ensures
semantic feature representation space consistency to fully distill the priors.
We demonstrate the effectiveness of our framework across multiple IR models and
tasks, including deraining, deblurring, and denoising.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要