An End to End Real Time Architecture for Analyzing and Clustering Time Series Data: Case of an Energy Management System

Big data is a field that fascinated many researchers from different areas to study intelligent and robust techniques to analyze extremely large data sets, reveal patterns about human behaviors and then make important decisions such as predicting the next human activity. Smart grid is a cyber physical system that aims at updating the current old-fashion electrical grid by incorporating the latest ICTs to improve the generation, the distribution and the consumption of electricity. The use of sensors in smart grids becomes crucial as it allows an energy management system to analyze massive generated sensory data and use machine learning algorithms to take advantage of the customer’s participation to reduce the cost of power. However, big data field revealed a very long list of tools to analyze data either in real time or batch modes so the decision of what tools to use for a particular case becomes a challenging one. The purpose of this paper is to present an end-to – end architecture for a real time test bed implemented at Al Akhawayn University in Ifrane Morocco to analyze and cluster time series sensor data using an IoT architecture composed of Kaa (a middleware), Kafka (a realtime data messaging queue), Spark (an in-memory data analytics platform) and k-means (a clustering algorithm).