The Fornax Deep Survey with the VST XII. Low surface brightness dwarf galaxies in the Fornax cluster

Context. Low surface brightness (LSB) dwarf galaxies in galaxy clusters are an interesting group of objects as their contribution to the galaxy luminosity function and their evolutionary paths are not yet clear. Increasing the completeness of our galaxy catalogs is crucial for understanding these galaxies, which have effective surface brightnesses below 23 mag arcsec(-2) (in optical). Progress is continuously being made via the performance of deep observations, but detection depth and the quantification of the completeness can also be improved via the application of novel approaches in object detection. For example, the Fornax Deep Survey (FDS) has revealed many faint galaxies that can be visually detected from the images down to a surface brightness level of 27 mag arcsec(-2), whereas traditional detection methods, such as using Source Extractor (SE), fail to find them. Aims. In this work we use a max-tree based object detection algorithm (Max-Tree Objects, MTO) on the FDS data in order to detect previously undetected LSB galaxies. After extending the existing Fornax dwarf galaxy catalogs with this sample, our goal is to understand the evolution of LSB dwarfs in the cluster. We also study the contribution of the newly detected galaxies to the faint end of the luminosity function. Methods. We test the detection completeness and parameter extraction accuracy of MTO using simulated and real images. We then apply MTO to the FDS images to identify LSB candidates. The identified objects are fitted with 2D Sersic models using GALFIT and classified as imaging artifacts, likely cluster members, or background galaxies based on their morphological appearance, colors, and structure. Results. With MTO, we are able to increase the completeness of our earlier FDS dwarf catalog (FDSDC) 0.5-1 mag deeper in terms of total magnitude and surface brightness. Due to the increased accuracy in measuring sizes of the detected objects, we also add many small galaxies to the catalog that were previously excluded as their outer parts had been missed in detection. We detect 265 new LSB dwarf galaxies in the Fornax cluster, which increases the total number of known dwarfs in Fornax to 821. Using the whole cluster dwarf galaxy population, we show that the luminosity function has a faint-end slope of alpha = -1.38 +/- 0.02. We compare the obtained luminosity function with different environments studied earlier using deep data but do not find any significant differences. On the other hand, the Fornax-like simulated clusters in the IllustrisTNG cosmological simulation have shallower slopes than found in the observational data. We also find several trends in the galaxy colors, structure, and morphology that support the idea that the number of LSB galaxies is higher in the cluster center due to tidal forces and the age dimming of the stellar populations. The same result also holds for the subgroup of large LSB galaxies, so-called ultra-diffuse galaxies.