On The Failure of Invariant Risk Minimization and an Effective fix via Classification Error Control

Invariant Risk Minimization is a well-known Domain Generalization framework that has received much attention over the past few years. Invariant Risk Minimization is capable of learning domain-invariant features from multiple domains by finding representation features such that the optimal classifier on top of these features matches all training domains. In this paper, we show that even though the Invariant Risk Minimization algorithm is based on a compelling idea, it is easily vulnerable in a simple toy example where multiple domain-invariant features exist and each possesses a corresponding classifier that is optimal for all domains. Based on this observation, we propose an effective modification of the traditional Invariant Risk Minimization algorithm named Error-Control Invariant Risk Minimization, which allows learning different domain-invariant features via controlling the training classification error, leading to a new algorithm that works well on both our toy synthetic dataset and the real-world datasets.