Reliable Accuracy Estimates from k-fold Cross Validation
IEEE Transactions on Knowledge and Data Engineering(2020)
摘要
It is popular to evaluate the performance of classification algorithms by
k
-fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform
k
-fold cross validation. Most of them did not consider the correlation among the replications of
k
-fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether
k
-fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of
k
-fold cross validation are first analyzed for
k
-nearest neighbors with
$k= 1$
. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of
k
-fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of
k
-fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The
k
-fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
更多查看译文
关键词
Reliability,Classification algorithms,Correlation,Statistical analysis,Testing,Roads,Urban areas
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要