Improving the efficacy of automated sign language practice tools

The CopyCat project is an interdisciplinary effort to create a set of computer-aided language learning tools for deaf children. The CopyCat games allow children to interact with characters using American Sign Language (ASL). Through Wizard of Oz pilot studies we have developed a set of games, shown their efficacy in improving young deaf children’s language and memory skills, and collected a large corpus of signing examples. Our previous implementation of the automatic CopyCat games uses sign language verification in the infrastructure of a memory repetition and phrase verification task. Sign language verification compares each input phrase to the correct phrase and accepts or rejects the sample. This approach only accepts language usage which is an exact match for the game phrases. The goal of my research is to expand the CopyCat system to use automatic sign language recognition and language processing to allow for more flexible language usage. I have created a labeling ontology from analysis of the CopyCat signing corpus, and I have used the ontology to describe the contents of the CopyCat data set. This ontology was used to improve automatic sign language recognition and to add a customized language processing component to the automatic game. Through these activities, I have created a automatic game component which combines automatic sign language recognition and language processing to enable dialogue-based interactions that better represent the usage of American Sign Language by the children in the CopyCat signing corpus.