GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory
CoRR(2024)
摘要
We present GenEFT: an effective theory framework for shedding light on the
statics and dynamics of neural network generalization, and illustrate it with
graph learning examples. We first investigate the generalization phase
transition as data size increases, comparing experimental results with
information-theory-based approximations. We find generalization in a Goldilocks
zone where the decoder is neither too weak nor too powerful. We then introduce
an effective theory for the dynamics of representation learning, where
latent-space representations are modeled as interacting particles (repons), and
find that it explains our experimentally observed phase transition between
generalization and overfitting as encoder and decoder learning rates are
scanned. This highlights the power of physics-inspired effective theories for
bridging the gap between theoretical predictions and practice in machine
learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要