Orama: A Benchmark Framework for Function-as-a-Service

The prominent Function-as-a-Service (FaaS) cloud service model has positioned itself as an alternative for solving several problems, and, interest in cloud-oriented architectural solutions that use FaaS has therefore grown rapidly. Consequently, the importance of knowing the behavior of FaaS-based architectures under different concurrency scenarios has also become significant, especially in implementation decision-making processes. In this work, the Orama framework is proposed, which helps in the execution of benchmarks in FaaS-based environments, orchestrating the deployment of pre-built architectures, as well as the execution of tests and statistical analysis. Experiments were carried out with architectures containing multiple cloud services in conjunction with FaaS in two public cloud providers (AWS and GCP). The results were analyzed using factorial design and t-test and showed that the use cases running on AWS obtained better results in runtime compared to their counterparts on GCP, but showed considerable error rates in competition situations. It is worth mentioning that the Orama framework was used from in the automated provisioning of use cases, execution of benchmarks, analysis of results and deprovisioning of the environment, supporting the entire process.