Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum
arxiv(2023)
摘要
Wide neural networks are biased towards learning certain functions,
influencing both the rate of convergence of gradient descent (GD) and the
functions that are reachable with GD in finite training time. As such, there is
a great need for methods that can modify this bias according to the task at
hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel
family of constructed kernels that can be used to approximate kernels with
desired eigenvalues for which no closed form is known. We leverage the duality
between wide neural networks and Neural Tangent Kernels and propose a
preconditioned gradient descent method, which alters the trajectory of GD. As a
result, this allows for a polynomial and, in some cases, exponential training
speedup without changing the final solution. Our method is both computationally
efficient and simple to implement.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要