A Multi-label Multimodal Deep Learning Framework for Imbalanced Data Classification

Social media and Web services have provided a notable number of multimedia content. Due to such explosion of multimedia data, the multimedia community has been facing new challenges and exciting opportunities these days. This paper presents a new multimedia framework to address some of the main challenges in this area. In particular, it presents a multi-label multimodal framework for imbalanced data classification. For this purpose, it utilizes audio, visual, and textual data modalities and automatically generates static and temporal features using spatio-temporal deep neural networks. It also manages data with non-uniform distributions using a weighted multi-label classifier. To evaluate this framework, a video dataset containing natural disasters is used for multi-label classification. The supremacy of the proposed framework compared to the existing work is revealed with extensive experiments on this dataset.