Robust nuclei segmentation in histopathology using ASPPU-Net and boundary refinement

Automated nuclear segmentation in histopathological images is a prerequisite for a computer-aided diagnosis framework. However, it remains a challenging problem due to the nucleus occlusion or overlapping, shape variation, and image background complexity. Recently, deep learning techniques are widely used in analyzing digital histopathology. We present a computerized image-based method for automatically segmenting nuclei using an integration of a deep learning model and an improved concave point detection algorithm. A modified atrous spatial pyramid pooling U-Net (ASPPU-Net) is derived to capture multi-scale nuclei features and obtain nuclei context information without reducing the spatial resolution of feature map. A weighted binary cross entropy loss function with Dice loss function is used to better handle the data unbalance problem. An accelerated concave point detection method allows to effectively and accurately segmenting highly overlapping nuclei. Our ASPPU-Net based method was tested on four independent data cohorts and achieved the highest Dice similarity coefficient of 0.83, and pixel wise accuracy of 0.95. The experimental results suggested that the combination of ASPPU-Net model and concave point detection method was able to gain improved performance in separating both isolated and touching clustered nuclei in histopathology.