Comparison of NoSQL Datastores for Large Scale Data Stream Log Analytics

With the advent of cyber-physical systems, industrial internet of things (IIoT) and industrial analytics numerous application scenarios have emerged where business and mission-critical decisions depend upon large scale analysis of data in form of sensor streams. However, large volumes of sensor stream data generated at high frequency pose substantial challenges for existing scalable data analysis techniques requiring the use of high-performance distributed datastores. This work covers in-depth performance comparison of three principal categories of distributed state-of the-art NoSQL datastores by evaluating their applicability and efficiency for large-scale analysis of sensor logs from real-world hydraulic power systems. One central datastore is selected from each of the three principal categories of NoSQL datastores: MongoDB from the document store, Cassandra from the column store and Redis from the distributed main memory key-value store to be included in the performance evaluation. Understanding the differences and behavior of this type of systems are crucial for optimizing application performance. Key insights from this work can serve as a basis for an improved understanding of the applicability of NoSQL datastores in systems for large scale data stream analysis. This will be important for supporting data analytics in IIoT applications as found in monitoring and control of Power plants, Smart Cities, Transportation systems, Environmental and Health monitoring, etc.