A Novel POS-Guided Data Augmentation Method for Sign Language Gloss Translation.

Due to its profound significance for the hearing-impaired community, there has been an abundance of recent research focused on Sign Language Translation (SLT), which is often decomposed into video-to-gloss recognition (S2G) and gloss-to-text (G2T) translation. Here, a gloss represents a sequence of transcribed spoken language words arranged in the order they are signed. In this paper, our emphasis lies in G2T, a crucial aspect of sign language translation. However, G2T encounters a scarcity of data, leading us to approach it as a low-resource neural machine translation (NMT) problem. Nevertheless, in contrast to traditional low-resource NMT, gloss-text pairs exhibit a greater lexical overlap but lower syntactic overlap. Hence, leveraging this characteristic, we utilize part-of-speech (POS) distributions obtained from numerous monolingual spoken language text to generate high-quality pseudo glosses. By simultaneously training numerous pseudo gloss-text sentence pairs alongside authentic data, we have significantly improved the translation performance of Chinese Sign Language (+3.10 BLEU), German Sign Language (+1.10 BLEU), and American Sign Language (+5.06 BLEU) in G2T translation, respectively.