The Decline of Disruption in Science and Technology∗

Although the number of new scientific discoveries and technological inventions has increased dramatically over the past century, there are growing concerns that progress is slowing. We analyze 25 million papers and 4 million patents across 6 decades and find that science and technology are becoming less disruptive of existing knowledge, a pattern that holds nearly universally across fields. We link this decline in disruptiveness to a narrowing in the utilization of existing knowledge. Diminishing quality of published science and changes in citation practices are unlikely to be responsible for this trend, suggesting that this pattern represents a fundamental shift in science and technology. ∗E-mail park1892@umn.edu, leahey@arizona.edu, or rfunk@umn.edu. We thank the National Science Foundation for financial support of work related to this project (grants 1829168 and 1932596). We would also like to thank Daniel McFarland, Staša Milojević, Raviv Murciano-Goroff, Jonathan O. Allen, the participants of the CADRE workshop, and the reviewers of the Academy of Management TIM division for their feedback. 1 ar X iv :2 10 6. 11 18 4v 3 [ cs .S I] 1 0 O ct 2 02 1 While the past century witnessed an unprecedented expansion of scientific and technological knowledge, there are concerns that innovative activity is slowing Jones [2009], Gordon [2016], Chu and Evans [2021]. Studies document declining research productivity in semiconductors, pharmaceuticals, and other fields Pammolli et al. [2011], Bloom et al. [2020]. Papers, patents, and even grant applications have become less novel and less likely to connect disparate areas of knowledge, both of which are precursors of innovation Packalen and Bhattacharya [2020], Jaffe and Lerner [2011]. The gap between the year of discovery and the awarding of a Nobel Prize has also increased [e.g., Horgan, 2015, Collison and Nielsen, 2018], suggesting that today’s contributions may not measure up to the past. Numerous explanations for this slowdown have been proposed. Some point to a dearth of “low hanging fruit,” as the easier innovations have already been produced Cowen [2011], Gordon [2016]. Others suggest the decline is due to an increasing burden of knowledge; scientists and inventors require more training to reach the frontier of their field, leaving less time for making breakthroughs [e.g., Jones, 2009]. Yet much remains unknown, not merely about the causes of slowing innovative activity, but also the depth and breadth of the phenomenon. To date, the evidence pointing to a slowdown is based on studies of particular fields, using disparate and domain-specific metrics Pammolli et al. [2011], Bloom et al. [2020], making it difficult to know whether the changes are happening at similar rates across areas of science and technology. Little is also known about whether the patterns seen in aggregate indicators may mask differences in the degree to which individual works push the frontier. We address these gaps in knowledge by analyzing 25 million papers (1945-2010) in the Web of Science (“WoS data”) and 4 million patents from (1976-2010) in the United States Patent and Trademark Office’s Patents View database (“USPTO data”). The WoS data include 159 million citations and 28 million paper titles and abstracts. The USPTO data include 18 million citations and 6 million patent titles and abstracts. Using these data, we join a novel citation-based measure Funk and Owen-Smith [2017] with textual analyses of titles and abstracts to understand whether papers and patents forge new directions over time and across fields. To characterize the nature of innovation, we draw on foundational theories of scientific and technological change Schumpeter [1942], which distinguish between two types of breakthroughs. First, some contributions improve existing streams of knowledge, and therefore consolidate the status quo. Kohn & Sham (1965) Kohn and Sham [1965], a Nobel-winning paper (“KS”) utilized established theorems to develop a method for calculating the structure of electrons, which cemented the value of prior research. Second, some contributions disrupt existing knowledge, rendering it obsolete, and propelling science and technology in new directions. Watson & Crick (1953) Watson and Crick [1953] (“WC”), also a Nobel winner, introduced a model of the structure of DNA that superseded previous approaches (e.g., Pauling’s triple helix). KS and WC were both important, but their implications for scientific and technological change were different. To quantify this distinction, we utilize a measure—the CD index—which characterizes the consolidating/disruptive nature of science and technology based on citation networks (fig. 1). The intuition is that if a paper or patent is disruptive, the subsequent work that cites it is less likely to also cite its predecessors; for future researchers, the ideas that went into its production are less relevant (e.g., Pauling’s triple helix). If a paper or patent is consolidating, subsequent work that cites it is also more likely to cite its predecessors; for future researchers, the knowledge upon which the work builds is still (and perhaps more) relevant (e.g., the theorems KS used). The CD index ranges from -1 (most consolidating) to 1 (most disruptive). We measure the CD index five years after the year of publication (indicated by CD5). For example, WC and KS both received over a hundred citations within five years of publication. However, the KS paper has a CD5 of -0.22 (indicating consolidation), whereas the WC paper has a CD5 of 0.62 (indicating disruption). The CD index has been validated in prior research, including with expert assessments Funk and Owen-Smith [2017], Wu et al. [2019]. Across fields, we find the rate of disruptive work is declining. Fig. 2 plots the average CD5 over time for papers (fig. 2A) and patents (fig. 2B). For papers, the decrease (1945-2010) ranges from 91.9% (Social Science) to 100% (Physical Science); for patents, the decrease (1980-2010) ranges from 93.5% (Computers and Communications) to 96.4% (Drugs and Medical). We verify the decline in disruptiveness over time through regression models which show that year has a statistically significant and negative relationship with CD5 for papers (p < 0.0001) and patents (p < 0.0001). These declines demonstrate that relative to earlier eras, recent papers and patents do less to push fields in new directions in a way that surpasses prior work. The similarity in trends we observe across fields is noteworthy in light of “low hanging fruit” arguments Cowen