Expand-and-Cluster: Parameter Recovery of Neural Networks
CoRR(2023)
摘要
Can we identify the parameters of a neural network by probing its
input-output mapping? Usually, there is no unique solution because of
permutation, overparameterisation and activation function symmetries. Yet, we
show that the incoming weight vector of each neuron is identifiable up to sign
or scaling, depending on the activation function. For all commonly used
activation functions, our novel method 'Expand-and-Cluster' identifies the size
and parameters of a target network in two phases: (i) to relax the
non-convexity of the problem, we train multiple student networks of expanded
size to imitate the mapping of the target network; (ii) to identify the target
network, we employ a clustering procedure and uncover the weight vectors shared
between students. We demonstrate successful parameter and size recovery of
trained shallow and deep networks with less than 10
number and describe an 'ease-of-identifiability' axis by analysing 150
synthetic problems of variable difficulty.
更多查看译文
关键词
exact parameter recovery,neural networks,expand-and-cluster
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要