Equivalence of the Empirical Risk Minimization to Regularization on the Family of f-Divergences
CoRR(2024)
摘要
The solution to empirical risk minimization with f-divergence
regularization (ERM-fDR) is presented under mild conditions on f. Under
such conditions, the optimal measure is shown to be unique. Examples of the
solution for particular choices of the function f are presented. Previously
known solutions to common regularization choices are obtained by leveraging the
flexibility of the family of f-divergences. These include the unique
solutions to empirical risk minimization with relative entropy regularization
(Type-I and Type-II). The analysis of the solution unveils the following
properties of f-divergences when used in the ERM-fDR problem: i)
f-divergence regularization forces the support of the solution to coincide
with the support of the reference measure, which introduces a strong inductive
bias that dominates the evidence provided by the training data; and ii)
any f-divergence regularization is equivalent to a different f-divergence
regularization with an appropriate transformation of the empirical risk
function.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要