Take a Close Look at the Optimization of Deep Kernels for Non-parametric Two-Sample Tests

The maximum mean discrepancy (MMD) test with deep kernel is a powerful method to distinguish whether two samples are drawn from the same distribution. Recent studies aim to maximize the test power of MMD test to find the best deep kernel for testing, where the test power is the ratio of MMD value and MMD's asymptotic variance. However, in this paper, we find that direct maximization of the test power sometimes leads to an unreasonable case that MMD value is very small but the test power is large. In this case, the testing performance is not satisfactory. Thus we propose two main methods to simultaneously maximize the test power and the MMD value with deep kernel by combining the methods from non-smooth optimization and the methods from Pareto optimization. Experiments verify the effectiveness of two methods on two benchmark datasets in the field of two-sample testing.