Testing Computational Assessment of Idea Novelty in Crowdsourcing

In crowdsourcing ideation websites, companies can easily collect large amount of ideas. Screening through such volume of ideas is very costly and challenging, necessitating automatic approaches. It would be particularly useful to automatically evaluate idea novelty since companies commonly seek novel ideas. Four computational approaches were tested, based on Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), term frequency - inverse document frequency (TF-IDF), and Global Vectors for Word Representation (GloVe), respectively. These approaches were used on three set of ideas and the computed idea novelty scores, along with crowd evaluation, were compared with human expert evaluation. The computational methods do not differ significantly with regard to correlation coefficients with expert ratings, even though TF-IDF-based measure achieved a correlation above 0.40 in two out of the three tasks. Crowd evaluation outperforms all the computational methods. Overall, our results show that the tested computational approaches do not match human judgment well enough to replace it. Plain Language SummaryCreative ideas are crucial to the success of organizations and the society in general. Information and communication technologies have made it possible to collect a large number of ideas in a short time period. However, evaluating and selecting among many ideas are demanding and time-consuming. Computational methods have been developed for processing textual documents. Yet there has been insufficient comparison of various methods in evaluating the novelty of ideas generated in crowdsourcing. This study compared four computational methods for evaluating the novelty of ideas in three distinct topics. Computationally evaluated novelty scores were compared to human evaluations. Our results show that the computationally generated scores were typically positively related to human judgment. However, computational methods did not match human evaluation close enough to replace it. Possible future developments were discussed.