A new method for building single feedforward neural network models for multivariate static regression problems: a combined weight initialization and constructive algorithm

The performance of models based on a Feedforward Neural Network depends strongly on the initial estimate of weights and the number of units in the hidden layers. This work presents a new method for the initialization of weights and definition of the number of hidden units in the identification of Multiple Input Single Output models associated with regression problems. The initialization strategy consists of a complete linearization of the network with only one neuron unit around an equilibrium point and the determination of the initial weights through the maximum approximation of the linearized model to the Optimal Linear Regressor whose solution can be obtained analytically. The constructive algorithm performs a gradual increase in the number of hidden units in such a way that at each training only weights associated with new hidden units are randomly initialized, while weights obtained from previous training are used as initial guess for the subsequent ones. The proposed method was compared to the classical random initialization method and to the Extreme Learning Machine (a typical gradient-free learning algorithm), both of which also involve a constructive approach to define the final network. The methods were applied to 11 real datasets widely used as benchmarks for regression problems. The proposed method performed better than the other approaches in 8–9 case studies and was able to ensure a monotonic decrease in the loss function with an increasing number of hidden units.