On obtaining sparse semantic solutions for inverse problems, control, and neural network training

•A novel alternative to PCA smoothly, monotonically converges to a regularized SVD.•A novel set of coordinate descent rules promotes sparser, interpretable solutions.•We test our methods by recovering parameters of a 3D facial model from a 2D RGB image.