Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.
EMNLP '08 Proceedings of the Conference on Empirical Methods in Natural Language Processing(2008)
摘要
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.
更多查看译文
关键词
Mechanical Turk non-expert annotation,non-expert contributor,non-expert label,Mechanical Turk system,affect recognition,annotation quality,faster method,gold standard annotation,gold standard label,human linguistic annotation,natural language task
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要