Lego: Dynamic Tensor-Splitting Multi-Tenant DNN Models on Multi-Chip-Module Architecture

Modern deep neural network (DNN) accelerators target the acceleration of a single DNN model and limit the throughput for multi-tenant DNN data center applications. The multi-chip-module (MCM) architecture breaks a monolithic accelerator into multiple small chiplets. The MCM is a promising approach that dispatches DNN models across chiplets with equal PEs. However, it is challenging to distribute data of DNN model layers with different parameters across chiplets while maximizing the chiplet utilization. This work proposes Lego MCM architecture that dynamically adapts to the size of DNN model layers and improves the throughput of multi-tenant DNN applications by increasing the chiplet utilization. Lego's dynamic scheduler achieves the geometric average 1.51× speedup over a monolithic DNN accelerator.