A Selective Comparative Review of CRISP-DM and TDSP Development Methodologies for Big Data Analytics Systems

Big data analytics systems (BDAS) are modern software systems with descriptive, predictive, or prescriptive purposes developed by current organizations. BDAS are viable due to the convergence of analytics techniques and the availability of sources of massive data, internal and external to the organization. BDAS are developed in a variety of domains of application such as marketing, healthcare, finance, manufacturing, logistics, education, and tourism, among others. However, although BDAS are modern software systems, organizations have used trial-and-error practical guidelines or old rigor-oriented heavyweight methodologies (a.k.a. plan-driven ones). The business competitive environment demands currently modern – i.e., lightweight or agile – BDAS development methodologies, and in the last years, the first modern methodologies have been proposed. However, studies contrasting rigor-oriented vs. lightweight or agile BDAS development methodologies are still scarce in the literature. In this chapter, we address this knowledge gap, and we report a comparative review between CRISP-DM – the main rigor-oriented BDAS methodology – and Team Data Science Process (TDSP), a new relevant proprietary agile one, by using a Scrum-XP workflow of practices as the theoretical agile development framework. Our comparative review provides theoretical and practical insights for discriminating both BDAS development approaches useful for researchers and practitioners in the BDAS development domain.