Evaluating Models with Dynamic Sampling Holdout in Auto-ML

Automated Machine Learning (Auto-ML) is a growing research area that is receiving great attention today. Multiple techniques have been developed to improve the automation process for the construction of machine learning pipelines, using diverse types of approaches and with relative success, but still, being far from solved. Much of this difficulty is due to the computational cost involved in the process, given that just evaluating a single machine learning solution can be expensive by itself. One of the reasons for this high cost is the widespread use of cross-validation as the evaluation method, which aims to reduce the occurrence of overfitting during model selection. Despite aiding with overfitting, cross-validation is a costly procedure that further increases the computational time required by the Auto-ML strategies. In this sense, this work revisits the Auto-CVE (Automated Coevolutionary Voting Ensemble) and proposes a new method for model evaluation: the dynamic sampling holdout. The main change from the traditional use of holdout is that it is conceived as a generational process, iteratively modifying the training and testing sets to renew the evaluations obtained periodically and prevent the search process from becoming guided for a long time by an incorrect evaluation. When compared to the regular Auto-CVE with cross-validation and the popular techniques Auto-SKLearn and TPOT (Tree-based Pipeline Optimization Tool), Auto-CVE with dynamic holdout shows competitive results in both predictive performance and computing time.