BERTPerf: Inference Latency Predictor for BERT on ARM big.LITTLE Multi-Core Processors

Hardware-aware Neural Architecture Search (NAS) and mapping & scheduling optimization methods are being used to find efficient implementations of computationally-intense language models such as BERT. This requires measuring real hardware inference latency: good design decisions simply cannot be made with proxy metrics such as FLOPs or the number of parameters. However, the time required to perform on-device latency measurements is prohibitive (e.g., a few days to a few weeks over the course of an optimization run). To address this, we present BERTPerf, a low-cost, highly-accurate method to predict the inference time of BERT on ARM big.LITTLE multi-core processors. BERTPerf exploits latency patterns at the layer-level to reduce on-device latency measurements, and captures the effect of caching and intermediate tensor allocations to reduce latency prediction error. BERTPerf reduces the maximum prediction error by 7–11% compared to the state-of-the-art, and requires 75% less on-device measurements compared to existing work at the same prediction error.