The costs of complex model optimization

Each data-driven action in data modelling consumes degrees of freedom, whether it concerns estimation of parameters, estimation of meta-parameters or selecting variables. By using a double cross validation approach for degrees of freedom calculation the costs for meta-parameter estimation and variable selection can be determined explicitly. The only assumptions are independent and identically distributed errors, which make the approach applicable to many predictive modelling techniques.