Towards solving overlapping nuclei segmentation: Sequential CNNs for one to many mapping of pixels to objects

Automated cell nucleus segmentation is key to quantifying cell features and functionality to identify the disease state of tissue and its likely biological future development. Despite considerable advances in automated segmentation, it is a challenging task to split overlapping clustered nuclei where pixels can belong to more than one nuclei. To substantially improve upon current cell/nuclei segmentation methods our group has developed a promising nuclear segmentation method using sequential rounds of modified UNet CNNs which we denote SUNet. This sequential application of two separate CNNs is required to allow the reparsing the original image into multiple overlapping sub images which explicitly allows for the SUNet approach to perform a one pixel mapping to many objects that is missing from other CNN based segmentation methods in Digital Pathology. We initially constructed two human annotated 8000 overlapping nuclei training sets one each from lung and prostate tissue sections quantitatively stained for DNA. These were used to train two SUNets. Their performance was evaluated on ~20,000 nuclei from multiple tissues for each tissue type. The average correct visual nuclei segmentation rate was observed to be ~84%. A subset of the training images were hand annotated by multiple observers and the Jaccard similarity coefficient (JSC) was used to quantify the similarity between these annotations. Four human observers have annotated the same 1065 nuclei from 100 nuclei clusters. For the 6 pairwise comparisons of these individuals, the average JSC was found to be 0.69 (range 0.63-0.75). This low number reflects the inability of humans to exactly trace the same boundary by hand and that not all observers recognized the same nuclei within the highly complex clusters. The SUNets JSC was similar to the average JSC and within the range of the human observers. Indicating that for these images the SUNet performs as well as a human at segmenting nuclei within complex clusters of nuclei. Comparing the performance of SUNet with a published mask_RCNN based nuclei segmentation and a recently published nucleAIzer method on a non-complex (not a lot of overlapping nuclei) image; we find the three are almost equivalent. However, the two published methods miss a few overlapping nuclei that are found by SUNet. On images with highly complex overlapping clusters of nuclei SUnet correctly segments more (10-20%, image complexity dependent) nuclei than the other methods in areas of high complexity. In addition, SUNet recognizes and segments large numbers of nuclei completely missed by the other methods. An added benefit of allowing a one to many assignment of pixels, is that it correctly preserves the complete shape of the nuclei as it is able to assign the same pixel(s) in an image to the multiple nuclei involved in the area of their overlap as apposed to assigning it to one of the nuclei involved. In summary, the sequential CNN proposed approach can segment nuclei in complex cluster with an accuracy equivalent to human observers. Citation Format: Calum E. MacAulay, Kouther Noureddine, Martial Guillaud, Paul Gallagher. Towards solving overlapping nuclei segmentation: Sequential CNNs for one to many mapping of pixels to objects [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-009.