A Large-Scale Combinatorial Benchmark for Sign Language Recognition

Lacking a large-scale dataset is the major obstacle limiting sign language recognition (SLR) to work well in the real world, because of the huge collection and annotation cost of sign language videos. This paper rethinks a sign language sentence as a combination of a template (T) and an entity (E) and presents a novel T and E Disassemble-and-reAssemble (TEDA) strategy to collect T and E sign videos independently. The proposed TEDA strategy has a theoretical capability of generating T x E effective samples with only T + E collection and annotation cost. With the TEDA strategy, we build a cost-controllable large-scale (CCLS) sign language dataset with 300,400 combinatorial samples, generated from 6,000 T videos and 29,700 E videos. To enable training arbitrary SLR models on combinatorial data, we propose a combinatorial SLR framework. Specifically, we first design a dynamic combination module to dynamically combine independent T and E features to generate combinatorial features. Then, we propose a joint constraint module to ensure that the distribution of the combinatorial features is as close as possible with the complete features. Finally, we develop a multi-stage training strategy to accommodate SLR learning with the combinatorial data. Plentiful experiments demonstrate the rationality of our TEDA strategy in generating large-scale effective combinatorial samples as well as the effectiveness of the combinatorial framework in promoting SLR.