Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicsEI

    Giancarlo Kerg
    Giancarlo Kerg
    Kyle Goyette
    Kyle Goyette
    Maximilian Puelma Touzel
    Maximilian Puelma Touzel
    Gauthier Gidel
    Gauthier Gidel
    Click here to see all papers in nips2019
    Cited by: 1|Bibtex|71|

    NeurIPS, 2019.

    Keywords:
    vanishing gradient problemschur decompositionorthogonal matricesschur form

    Abstract:

    A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the co...More
    Your rating :
    0

     

    Tags
    Comments