Pseudo inverse versus iterated projection: Novel learning approach and its application on broad learning system

Broad learning system (BLS) has attracted widespread attention owing to its concise structure and efficient incremental learning based on ridge regression approximation of pseudo-inverse. However, BLS struggles with expensive computational costs and high memory usage when processing complex problems and training large networks due to the inverse operation of a large matrix. There has been some attempts to address this issue. In this work, we attempt to ameliorate the computational efficiency motivated by the state-of-the-art randomized iterative least squares solver. First, we improve the iteration manner of randomized extended Kaczmarz (REK) and then propose the iterated projection learning for training BLS without pseudo-inverse. The iterated projection learning is suitable for both offline and online learning scenarios and can adjust the model structure by incremental learning. Thanks to its linear convergence of least norm in expectation, the obtained model is expected to have better generalization ability without additional regularization procedure. Our model is found tomaintain stable performance during incremental learning. Moreover, instead of grid search, we present an evolutionary bilevel programming (EBP) method with dispersion operator to further optimize the hyperparameters of the network structure. Numerical experiments indicate that the proposed methods can improve the efficiency, robustness, and generalization ability compared with pseudo-inverse and grid search based methods.