Tri-Branch CNN for Age-Related Macular Degeneration Categorization with Incomplete Multi-Modality Ophthalmology Images

The problems of incomplete data and insufficient feature utilization are commonly existing in disease diagnosis with multi-modality ophthalmology images. In this paper, we firstly develop a three-constraint generative adversarial network (TCGAN) to synthesize the missing ophthalmology images, so that the differences between the feature maps of the synthesized images and the corresponding real images are as small as possible to preserve the information relevant to the diagnosis. Then, we propose a CFP-OCT multi-modality network (COMNet), in which two branches are used to extract features from CFP images and OCT images, and a fusion branch is composed of four multi-modality fusion (MMF) modules, which adopts an attention mechanism based fusion method to generate delicate fusion features at each stages of the network. Experimental results showed that using TCGAN to synthesize missing images for data augmentation is effective, while COMNet achieved better diagnosis performance.