Anomaly-Aware 3D Segmentation of Knee Magnetic Resonance Images

In medical imaging, anatomical structures under examination often contain anomalies or pathologies making automated segmentation challenging in these situations. Hence, the robust segmentation of anatomical structures in the presence of anomalies represents an important step within the medical image analysis field. In this work, we show how popular U-Net-based neural networks can be used for detecting anomalies in the knee from 3D magnetic resonance (MR) images in patients with varying grades of osteoarthritis (OA). We also show that the extracted information can be utilized for downstream tasks such as parallel segmentation of anatomical structures along with associated anomalies such as bone marrow lesions (BMLs). For anomaly detection, a U-Net-based model was adopted to inpaint the region of interest in images so that the anomalous regions can be replaced with close to normal appearances. The difference between the original image and the inpainted image was then used to highlight the anomalies. The extracted information was then used to improve the segmentation of bones and cartilages; in particular, the anomaly-aware segmentation mechanism provided a significant reduction in surface distance error in the segmentation of knee MR images containing severe anomalies within the distal femur.