Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation
arXiv (Cornell University)(2021)
摘要
Human annotations play a crucial role in machine learning (ML) research and
development. However, the ethical considerations around the processes and
decisions that go into building ML datasets has not received nearly enough
attention. In this paper, we survey an array of literature that provides
insights into ethical considerations around crowdsourced dataset annotation. We
synthesize these insights, and lay out the challenges in this space along two
layers: (1) who the annotator is, and how the annotators' lived experiences can
impact their annotations, and (2) the relationship between the annotators and
the crowdsourcing platforms and what that relationship affords them. Finally,
we put forth a concrete set of recommendations and considerations for dataset
developers at various stages of the ML data pipeline: task formulation,
selection of annotators, platform and infrastructure choices, dataset analysis
and evaluation, and dataset documentation and release.
更多查看译文
关键词
dataset annotation,collective identities
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要