A survey of transformer-based multimodal pre-trained modals

•Multimodal Pre-trained models with document layout, vision-text and audio-text domains as input.•Collection of common multimodal downstream applications with related datasets.•Modality feature embedding strategies.•Cross-modality alignment pre-training tasks for different multimodal domains.•Variations of the audio-text cross-modal learning architecture.