SaPus: Self-Adaptive Parameter Update Strategy for DNN Training on Multi-GPU Clusters

Parameter server architecture has been identified as an efficient framework for scaling DNNs training on clusters. For large-scale deployment, communication becomes the bottleneck, and the parameter updating strategy strongly impacts the training performance and accuracy. Recent state-of-art solutions have adopted the local SGD approach, which enables workers to update their local version of model...