Exploring the Effect of ℓ0/ℓ2 Regularization in Neural Network Pruning using the LC Toolkit

The LC Toolkit is an open-source library written in Python and PyTorch that allows to compress any neural network using several compressions including quantization, pruning, and low-rank. The versatility of the framework is rooted in the principled mathematical formulation of the underlying network compression problems with subsequent optimization by learning-compression (LC) algorithm. In this paper, we utilize the LC toolkit’s common algorithmic base to take a deeper look into ℓ 0 -constrained pruning problems defined as follows: given a budget of κ non-zero weights, which weights should be pruned in the final network? We observe that ℓ 0 -pruned networks have a different connectivity structure compared to pruning results using ℓ 1 norm. We propose a change to the formulation of the problem involving a small amount of ℓ 2 weight decay which has a favorable effect on connectivity structure. We study the properties of the proposed ℓ 0 + ℓ 2 formulation using the LC toolkit and empirically demonstrate that such a scheme achieves a competitive sparsity-error tradeoff while having better structural sparsity.