Duplicate Identification Algorithms in SaaS Platforms

Existing duplicate records is one of the most common issues in many Software-as-as-Service (SaaS) platforms. In this paper, we study the duplicate identification problem in one specific SaaS platform related to quality and compliance management by using the address information. We interpret all typical mistakes from users that can generate the existent duplicated organizations in a given dataset, collected from the SaaS platform. Also, we create another set by crawling location data from Open Address (US Zone). We compare different methods, including Bag-of-words (using Cosine Distance), Record Linkage Toolkits, and Siamese Neural Networks using the triplet loss, in terms of precision, recall, and F1-score. The experimental results show that using Siamese Neural Networks can achieve a better performance in comparison with other techniques. We plan to publish our Open Address dataset and all implementation codes to facilitate further research in the related fields.