Estimating Link Confidence for Human-in-the-loop Table Annotation

Annotating tables with high-quality semantic labels and linking cells in these tables to entities in reference knowledge graphs (KGs) serve several downstream applications such as knowledge graph construction, data enrichment, data analytics, and so on. There is an increasing interest in combining algorithms for the automatic annotation of tables with interactive applications that let users control and improve the annotations and use them for data enrichment. In this paper, we propose a supervised entity linking network that behaves as a re-ranker and score normalizer of scores returned by off-the-shelves entity retrieval systems. By combining these refined scores with additional signals of reliability of the ranking obtained thereof, we can estimate a confidence score, which tells how reliable is the decision to link a cell to the top-ranked candidate entity. The confidence score is used to decide whether to establish a link or not and to prioritize the cells to be revised by the users to improve the annotation quality. Experiments suggest that the proposed approach provides an effective approach to identifying most critical decisions and supporting link revision within a human-in-the-loop data enrichment paradigm.