Fashion Item Captioning Via Grid-Relation Self-Attention and Gated-Enhanced Decoder

Fashion Item Captioning aims to describe fine-grained item details according to several image angles, which is an advancement of the conventional image captioning task(generating a simple sentence for a single image). Most recent researches are dedicated to describe one image, and less attention is paid to capturing item details from different angles. As a result, they commonly take advantage of formula structure in the general domain rather than taking into account the characteristics of product images in the fashion domain. In this paper, we re-define the fashion captioning task to be Fashion Item Captioning , which is aimed to describe the item angles based on a multi-branch design. Based on this thinking, fashion item captioning still face two challenges. First, existing image captioning methods simply consider the grid features as a set of visual tokens while ignoring the positional relationships among them. And these rich relationships are difficult to be established by such coarse-grained visual representations. To this end, we propose a Grid-relation Self-Attention(GSA) , in which three positional relations among grid-level features are captured to strengthen the visual representations from multi perspectives. Second, the attributes of products are usually scattered in images from different angles, which means different angles contribute differently to the sentence caption. Thus, a Gated-Enhanced Decoder(GED) is introduced to dynamically measure the contribution of different views to the target word. Finally, we apply GSA and GED to the vanilla transformer model for the fashion item captioning task. Extensive experiments demonstrate the proposed GSA-GED is effective. More remarkably, GSA-GED achieves competitive performance on Fashion-Gen and FACAD datasets, with the CIDEr-D score being increased from 106.7 % to 112.1 % , 47.1 % to 49.3 % , respectively.