Prognostic Analysis of Histopathological Images Using Pre-Trained Convolutional Neural Networks

Background: Histopathological images contain rich phenotypic descriptions of the molecular processes underlying disease progression. Convolutional neural networks (CNNs), a state-of-the-art image analysis technique in computer vision, automatically learns representative features from such images which can be useful for disease diagnosis, prognosis, and subtyping. Despite hepatocellular carcinoma (HCC) being the sixth most common type of primary liver malignancy with a high mortality rate, little previous work has made use of CNN models to delineate the importance of histopathological images in diagnosis and clinical survival of HCC.Results: We applied three pre-trained CNN models -- VGG 16, Inception V3, and ResNet 50 -- to extract features from HCC histopathological images. The visualization and classification showed clear separation between cancer and normal samples using image features. In a univariate Cox regression analysis, 21.4% and 16% of image features on average were significantly associated with overall survival and disease-free survival, respectively. We also observed significant correlations between these features and integrated biological pathways derived from gene expression and copy number variation. Using an elastic net regularized CoxPH model of overall survival, we obtained a concordance index (C-index) of 0.789 and a significant log-rank test (p = 7.6E−18) after applying Inception image features. We also performed unsupervised classification to identify HCC subgroups from image features. The optimal two subgroups discovered using Inception image features were significantly associated with both overall (C-index = 0.628 and p = 7.39E−07) and disease-free survival (C-index = 0.558 and p = 0.012). Our results suggest the feasibility of feature extraction using pre-trained models, as well as the utility of the resulting features to build an accurate prognosis model of HCC and highlight significant correlations with clinical survival and biological pathways. Conclusions: The image features extracted from HCC histopathological images using the pre-trained CNN models VGG 16, Inception V3 and ResNet 50 can accurately distinguish normal and cancer samples. Furthermore, these image features are significantly correlated with relevant biological outcomes.