Rethinking the disruption index as a measure of scientific and technological advances

Wu et al. (2019) used the disruption(D) index to measure scientific and technological advances in Nature. Their findings spurred extensive discussion in academia on whether we can measure the disruption (i.e., innovation or novelty) of a research paper or a patent based on the number of citations. In this paper, we calculate the D index of similar to 0.76 million publications published between 1954 and 2013 in six disciplines including both sciences and social sciences in English and Chinese. We found that the number of references has a negative effect on the D index of a paper with a relatively small number of references, and a positive effect on the D index of a paper with a large number of references. We also found that low coverage of a citation database boosts D values. Specifically, low coverage of non-journal literature in the Web of Science (WOS) boosted D values in social sciences, and the exclusion of non-Chinese language literature in the Chinese Social Sciences Citation Index (CSSCI) resulted in the inflation of D values in Chinese language literature. Limitations of the D index observed in scientific papers also exist in technological patents. This paper sheds light on the use of citation-based measurements of scientific and technological advances and highlights the limitations of this index.