Contextualizing Lung Nodule Malignancy Predictions with Easy vs. Hard Image Classification.

In machine learning (ML), knowing the hardness of individual instances can provide multiple forms of utility to a model and its users. Designing a curriculum that presents training batches of gradually increasing hardness can improve generalization of the model. Furthermore, informing the user of the instance-level hardness along with a prediction can convey the model’s confidence for that prediction. Practically, providing the user with the hardness of a case can also assist with resource allocation. For example, in the medical domain, if a doctor knows that a case is easy, they may be able to proceed without calling a consult or ordering additional testing. In this work, we introduce a new approach to train an easy/hard classifier to predict hardness of unseen cases. As a proof of concept in the medical imaging domain, we use the National Institutes of Health National Cancer Institute Lung Image Database Consortium (NIH NCI LIDC) dataset to train a Convolutional Neural Network (CNN) in predicting hardness of lung nodule images with respect to the malignancy classification task. Despite a relatively small training set, our classifier achieves 86% testing accuracy. We then demonstrate how the learned hardness labels correspond to prediction confidence and radiologist-perceived hardness. Confidence is estimated using the softmax output of the malignancy prediction and radiologist-perceived hardness is evaluated using inter-rater agreement of malignancy ratings. This hardness prediction can be used to inform users about the confidence of a Computer-Aided Diagnosis (CAD) prediction or help generate easy and hard training sets for either human or machine curriculum learning (CL). While this work focuses on medical imaging applications, this methodology can be generalized to any domain to provide hardness ratings for unseen and unlabeled data.