Using principles of optimal treatment regime estimation in simulation studies for benchmarking

In benchmarking studies with simulated data sets in which two or more methods are compared, over and above the search of a universally winning method, one may investigate how the winning method may vary over patterns of characteristics of the data and the data-generating mechanism. Interestingly, this problem bears strong formal similarities to the problem of looking for optimal treatment regimes in biostatistics when two or more treatment alternatives are available for the same medical problem or disease. In that case, one may wish to induce from empirical data a rule that indicates which types of patients should preferably receive which treatment. The optimal rule or treatment regime then is the one that yields the highest expected (potential) outcome if the rule would be applied to the entire population of patients under study. In this talk, we will outline how rules for optimal calling in methods can be derived from benchmarking studies with simulated data by means of a classification tree method that is based on principles of optimal treatment regime estimation. We will illustrate by means of analyses of data from a benchmarking study to compare two different algorithms for the estimation of a two-mode additive clustering model.