Flexi-BOPI: Flexible Granularity Pipeline Inference with Bayesian Optimization for Deep Learning Models on HMPSoC

To achieve high-throughput deep learning (DL) model inference on heterogeneous multiprocessor systems-on-chip (HMPSoC) platforms, the use of pipelining for the simultaneous utilization of multiple resources has emerged as a promising solution. Nevertheless, current research faces two primary challenges: determining the optimal pipeline partitioning granularity, which directly influences the inference performance, and addressing the high time overhead of the search algorithms. To address these challenges, we propose Flexi-BOPI, a pipeline inference method for DL models of HMPSoCs. Flexi-BOPI offers flexible pipeline partitioning granularity down to a minimum size of a single core, enhancing the performance by better adapting to the diverse computational demands of different layers in DL models. Flexi-BOPI employs a Bayesian optimization-based search algorithm to significantly reduce the search overhead. In addition, we propose a surrogate model based on the heteroscedastic Gaussian process (HGP) to address the challenge of sample noise during the evaluation process. This approach can further reduce search overhead. Our experimental results demonstrate that the proposed method achieves significant improvements in inference performance and search overhead compared to existing methods.