On-Premises Serverless Computing for Event-Driven Data Processing Applications

The advent of open-source serverless computing frameworks has introduced the ability to bring the Functions-as-a-Service (FaaS) paradigm for applications to be executed on-premises. In particular, data-driven scientific applications can benefit from these frameworks with the ability to trigger scalable computation in response to incoming workloads of files to be processed. This paper introduces an open-source framework to achieve on-premises serverless computing for event-driven data processing applications that features: i) the automated provisioning of an elastic Kubernetes cluster that can grow and shrink, in terms of the number of nodes, on multi-Clouds; ii) the automated deployment of a FaaS framework together with a data storage back-end that triggers events upon file uploads; iii) a service that provides a REST API to orchestrate the creation of such functions and iv) a graphical user interface that provides a unified entry point to interact with the aforementioned services. Together, this provides a framework to deploy a computing platform to create highly-parallel event-driven file-processing serverless applications that execute on customized runtime environments provided by Docker containers that run on an elastic Kubernetes cluster. The usefulness of this framework is exemplified by means of the execution of a data-driven workflow for optimised object detection on video. The workflow is tested under three different workloads which process ten, a hundred and a thousand functions. The results show that the presented architecture is able to process such workloads taking advantage of its elasticity to make a sensible usage of the resources.