PROMENADE: A big data platform for handling city complex networks with dynamic graphs

Continuous data streams, generated by modern sensed cities, open many opportunities and perspectives in terms of developing new innovative services. To exploit this potential, flexible and scalable platforms are needed to ease the design, development, deployment, and operations of new city services. In recent years, several problem-specific platforms have been proposed in different application domains; however, to boost the evolution of smart cities, we claim the need for city-oriented platforms that can be easily customized to address different day-to-day life challenging problems. In this paper, we present the main architectural challenges and solutions proposed for the design of a novel open-source platform (named PROMENADE) characterized by: i) a data-driven graph-based modeling support to ensure high generality for addressing disparate problems related to the networked nature of many city infrastructures and systems, ii) the dynamic nature of the graph entities updated in real-time from different sources (e.g., IoT/Edge networks, data providers, etc.), and iii) high efficiency, scalability and flexibility to easily support new city services. The platform is designed around a general-purpose core that provides a set of built-in standard features such as data ingestion, storage, processing, and visualization exposed as a collection of containerized microservices. A specialization of the platform has been developed for road networks monitoring. It has been deployed in OpenShift/Kubernetes and tested using realistic datasets collected from the city of Lyon, France. The analysis addresses an important problem of big data processing pipelines: the synchronization between data ingestion and processing in order to produce an accurate result in useful time. To this end, we study different approaches for synchronization and show how the end-to-end latency is kept under control by leveraging the scalability of the platform.