MDDRSPF: A Model Driven Distributed Real-Time Stream Processing Framework

Some enterprises have demands for distributed stream processing framework to manage and process diverse business services. But their business is usually user-oriented, the access data sources and outputs are heterogeneous and need to be configurable, and the user-oriented service mode requires the platform have the ability to deploy new algorithm dynamically and provide on-demand services. As we know, there is no system that can meet all these requirements, hence a Model Driven Distributed Real-time Stream Processing Framework is proposed. By building project and task models, the functional model is separated from the implementation platform. The framework support multi-source heterogeneous data input and configurable output, on-demand services based on user's request, and algorithm dynamic loading. Service oriented architecture is deployed to lower the threshold of algorithm development and deployment. Furthermore, the functionality and performance is evaluated with a practical case to verify whether the system can meet the needs of practical application.