How to train your neural ODE: the world of Jacobian and kinetic regularization
international conference on machine learning(2020)
摘要
Training neural ODEs on large datasets has not been tractable due to the
necessity of allowing the adaptive numerical ODE solver to refine its step size
to very small values. In practice this leads to dynamics equivalent to many
hundreds or even thousands of layers. In this paper, we overcome this apparent
difficulty by introducing a theoretically-grounded combination of both optimal
transport and stability regularizations which encourage neural ODEs to prefer
simpler dynamics out of all the dynamics that solve a problem well. Simpler
dynamics lead to faster convergence and to fewer discretizations of the solver,
considerably decreasing wall-clock time without loss in performance. Our
approach allows us to train neural ODE-based generative models to the same
performance as the unregularized dynamics, with significant reductions in
training time. This brings neural ODEs closer to practical relevance in
large-scale applications.
更多查看译文
关键词
kinetic regularization,neural ode,jacobian
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要