Towards optimization of the gated recurrent unit (gru) for regression modeling

Deep Machine Learning takes place by adjusting the weights of deep neural networks – brain-like computational structures in order to optimize a given cost function. There exists a deep learning task where every neural model typically performs better. This paper explores optimization strategies for the GRU neural network such as dropout, gradient clipping and stacking ensembles of neural layers amongst others. The Recurrent Neural Networks (RNNs) are suited for classification and prediction problems based on time-series data. It has proven a challenging task for an ordinary RNN to learn long sequences leading to the invention of two other variants: Long Short Term Memory (LSTM) and the Gated Recurrent Unit (GRU). Whereas both of these compare well on most learning tasks, the structure of a LSTM is more complex with three gates and in effect has to compute and keep the cell state (Ct). This makes GRU simpler to implement and typically converges faster making it more appropriate for online learning. The results of experiments indicate an improved convergence rate and better performance guarantees of the GRU over LSTM for typical regression tasks.