Multi-modality fusion coupled with deep learning for improved outcome prediction in head and neck cancer

Multi-level multi-modality fusion radiomics is a promising technique with potential to improve the prognostication of cancer. We aim to use advanced fusion techniques on PET and CT images coupled with deep learning (DL) to improve outcome prediction in head and neck squamous cell carcinoma (HNSCC). In our study, 408 HNSCC patients were included from The Cancer Imaging Archive (TCIA) in a multi-center setting. Prognostic outcomes (binary classification) included overall survival (OS), distant metastasis (DM), locoregional recurrence (LR), and progression free survival (PFS). We utilized a DL algorithm with a 17-layer 3D convolutional neural network (CNN) architecture. Prior to training, each image underwent min-max-normalization, image-augmentation by using random rotations (0-20 degrees) to improve the performance and generalizability of our model and followed by 5-fold-cross-validation. We employed 12 datasets, including CT, PET, and 10 image-level fused datasets. The best OS performance was achieved via Discrete-wavelet-transform (DWT) resulting in mean accuracy of 0.93 +/- 0.06. The best DM score was achieved via ratio of low-pass pyramid (RLPP), resulting in an accuracy of 0.95 +/- 0.02. Optimal LR and PFS scores were achieved using DWT and RLPP for LR, and Laplacian pyramid for PFS, resulting in accuracies of 0.90-0.92. Comparatively, when using a machine learning framework instead of deep learning, we obtained scores of 0.83, 0.90, and 0.87 for the prediction of OS, DM, and LR. Our study demonstrates that our multi-modality fusion techniques performed better than using standalone PET or CT in prognostication of HNSCC patients and that a high level of accuracy can be achieved for prognostication of HNSCC patients when combining multi-modality fusion techniques with DL.