Container-Based Virtualization for Heterogeneous HPC Clouds: Insights from the EU H2020 CloudLightning Project

Building and successfully deploying applications on high-end heterogeneous resources such as GPUs, MICs or FPGAs, and typically with several library dependencies, is a complex task. Modern containerization provides a lightweight virtualization environment which can help solve some of these complex deployment and execution issues. By using containers, the software can be packaged with all the dependencies and tested in a single environment and deployable on heterogeneous architectures, easily. In this paper, we present our experiences with the container-based virtualized solutions that we developed for the use case applications in the EU H2020 project CloudLightning. We present specifics on their management and orchestration with specific software on the heterogeneous resources of our test-bed. The use cases include a Genomics application targeting FPGA-based DFEs, an Upscaling application for reservoir modeling, a ray tracing application targeting the MIC (Intel Xeon Phi co-processor), and a BLAS application with libraries, optimized for both CPU and GPU. An overview of the CloudLightning project and how the use case applications have been developed to be used as cloud services in the self-organizing, self-managing cloud technology, is also included.