Universal Approximation of Linear Time-Invariant (LTI) Systems through RNNs: Power of Randomness in Reservoir Computing
IEEE Journal of Selected Topics in Signal Processing(2023)
摘要
Recurrent neural networks (RNNs) are known to be universal approximators of
dynamic systems under fairly mild and general assumptions. However, RNNs
usually suffer from the issues of vanishing and exploding gradients in standard
RNN training. Reservoir computing (RC), a special RNN where the recurrent
weights are randomized and left untrained, has been introduced to overcome
these issues and has demonstrated superior empirical performance especially in
scenarios where training samples are extremely limited. On the other hand, the
theoretical grounding to support this observed performance has yet been fully
developed. In this work, we show that RC can universally approximate a general
linear time-invariant (LTI) system. Specifically, we present a clear signal
processing interpretation of RC and utilize this understanding in the problem
of approximating a generic LTI system. Under this setup, we analytically
characterize the optimum probability density function for configuring (instead
of training and/or randomly generating) the recurrent weights of the underlying
RNN of the RC. Extensive numerical evaluations are provided to validate the
optimality of the derived distribution for configuring the recurrent weights of
the RC to approximate a general LTI system. Our work results in clear signal
processing-based model interpretability of RC and provides theoretical
explanation/justification for the power of randomness in randomly generating
instead of training RC's recurrent weights. Furthermore, it provides a complete
optimum analytical characterization for configuring the untrained recurrent
weights, marking an important step towards explainable machine learning (XML)
to incorporate domain knowledge for efficient learning.
更多查看译文
关键词
Reservoir computing,echo state network,neural network,deep learning,system identification and approximation,explainable machine learning
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要