Social Relationship Recognition Based on Relational Self-Attention Mechanism

Social relations are closely related to each of us and are a crucial part of society. Recognizing the social relationships of people in pictures can improve AI’s understanding of human behavior, thereby facilitating collaborative interactions between computers and humans. Previous work only focused on a single picture, so too little information can be obtained. In this paper, we proposed Picture Reasoning Model(PRM) to achieve relationship classification, which innovatively uses the self-attention method to learn the association between relationships. The association between relationships is at the social level, thus using it to assist relationship recognition can get rid of the problem of insufficient information in a single picture. In addition, the model also adopts a two-stream approach, extracting both characters and global features for getting multiple perspectives information. We conduct extensive experiments on two benchmark datasets PIPA and PISC. Experimental results show that our model has improved the accuracy metric of the datasets compared with SOTA. On the PIPA dataset, the accuracy increases from 64.4% to 65.6%, and on the PISC dataset, the mAP raises from 72.7% to 73.2%, which validates the effectiveness of our proposals.