Efficient Network Compression Through Smooth-Lasso Constraint

The powerful capabilities of deep convolutional neural networks make them useful in various fields. However, most edge devices are difficult to afford the huge amount of parameters and high computational cost. Therefore, it is highly imperative to compress these huge models to make them lightweight to enable real-time inference on edge devices. Channel pruning is a mainstream method of network compression. Generally, the Lasso constraint is imposed on the scaling factor in the batch normalization layer to make them tend to zero for selecting unimportant channels and then prune them. However, Lasso is a non-smooth function that is not derivable at zero, we experimentally find that when the value of the loss function is small, it is difficult to decline continuously. Aiming at the above problems, this paper proposes a pruning strategy based on the derivable function Smooth-Lasso, using Smooth-Lasso as a regularization constraint to perform sparse training and then prune the network. Experiments on benchmark datasets and convolutional networks show that our method can not only make the loss function converge quickly, but also save more storage space and computational cost than the baseline method while maintaining the same level of accuracy as the original network.