HViT: Hybrid vision inspired transformer for the assessment of carotid artery plaque by addressing the cross-modality domain adaptation problem in MRI.

BACKGROUND:Medical image classification is crucial for accurate and efficient diagnosis, and deep learning frameworks have shown significant potential in this area. When a general learning deep model is directly deployed to a new dataset with heterogeneous features, the effect of domain shifts is usually ignored, which degrades the performance of deep learning models and leads to inaccurate predictions. PURPOSE:This study aims to propose a framework that utilized the cross-modality domain adaptation and accurately diagnose and classify MRI scans and domain knowledge into stable and vulnerable plaque categories by a modified Vision Transformer (ViT) model for the classification of MRI scans and transformer model for domain knowledge classification. METHODS:This study proposes a Hybrid Vision Inspired Transformer (HViT) framework that employs a convolutional layer for image pre-processing and normalization and a 3D convolutional layer to enable ViT to classify 3D images. Our proposed HViT framework introduces a slim design with a multi-branch network and channel attention, improving patch embedding extraction and information learning. Auxiliary losses target shallow features, linking them with deeper ones, enhancing information gain, and model generalization. Furthermore, replacing the MLP Head with RNN enables better backpropagation for improved performance. Moreover, we utilized a modified transformer model with LSTM positional encoding and Golve word vector to classify domain knowledge. By using ensemble learning techniques, specifically stacking ensemble learning with hard and soft prediction, we combine the predictive power of both models to address the cross-modality domain adaptation problem and improve overall performance. RESULTS:The proposed framework achieved an accuracy of 94.32% for carotid artery plaque classification into stable and vulnerable plaque by addressing the cross-modality domain adaptation problem and improving overall performance. CONCLUSION:The model was further evaluated using an independent dataset acquired from different hardware protocols. The results demonstrate that the proposed deep learning model significantly improves the generalization ability across different MRI scans acquired from different hardware protocols without requiring additional calibration data.