Rethinking grammatical error annotation and evaluation with the Amazon Mechanical Turk
IUNLPBEA '10 Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications(2011)
摘要
In this paper we present results from two pilot studies which show that using the Amazon Mechanical Turk for preposition error annotation is as effective as using trained raters, but at a fraction of the time and cost. Based on these results, we propose a new evaluation method which makes it feasible to compare two error detection systems tested on different learner data sets.
更多查看译文
关键词
pilot study,present result,trained raters,new evaluation method,error detection system,amazon mechanical turk,different learner data set,preposition error annotation,grammatical error annotation
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络