Improving Job Launch Rates in the TaPaSCo FPGA Middleware by Hardware/Software-Co-Design

In recent years, FPGAs have established themselves as an important acceleration platform next to GPUs in heterogeneous HPC systems, providing flexibility and high performance for tasks such as machine learning inference or DNA sequencing. While the design of the FPGA-based accelerator cores has become accessible to a broader range of users through customized RISC-V soft-cores and the maturity of High-Level Synthesis (HLS), the integration of and interaction with such accelerator cores in the overall heterogeneous system remains a challenging task. The open-source TaPaSCo framework eases this task by providing a concise software API and middleware for the interaction with FPGA-based accelerator system-on-chips automatically generated from user-provided accelerator cores. In this work, we present an extension of the TaPaSCo framework which improves the launch rates and latencies of FPGA-accelerated compute jobs, a crucial factor for the performance of the overall system, through hardware/software-co-design of an improved Rust-based software runtime, and a job dispatcher itself accelerated by hardware. Our evaluation shows that the new dispatchers can provide an improvement of up to 6x in job throughput with only minimal resource overhead.