On the automatic quality assessment of annotated sample data for object extraction from remote sensing imagery

Nowadays, the tasks of object extraction from remotely sensed imagery, such as built-up area, water-body, and other Earth surface information extraction, heavily rely on annotated sample data due to the wildly used machine learning algorithms, especially Deep Learning (DL) methods. Most algorithms, however, treat image-based annotations as ground truth and ignore the impact of unreliable annotations, leading to low-reliability solutions. Additionally, manual Annotation Quality aSsessment (AQS) of remote sensing data usually entails a significant workload in actual dataset production, particularly for large datasets. To alleviate these issues, we propose a novel Annotation Quality aSsessment Network (AQSNet) for the automatic quality assessment of remote sensing annotated samples. It uses a multi-scale channel-spatial attention module to simulate how an inspector examines annotated samples at local and global scales, enhancing the ability to identify poor-quality annotations. Besides, we present a simple method for generating effective training samples by analyzing the types and characteristics of low-quality annotated sample data. Furthermore, we introduce and make publicly available HBD41, a massive land cover dataset, with 1,945,034 samples of size 512 x 512 pixels. To validate our methodology, two AQS datasets are constructed with HBD4 and public building datasets. On the building AQS dataset, the proposed method outperforms the baseline model (HRNet) by 7.115% in IoU and 5.712% on the water-body AQS dataset, respectively. Extensive ablation and comparison experiments also demonstrate that the proposed approach can effectively assess the quality of remote sensing annotated samples. The high-quality sample data acquired by our AQSNet serves as a pre-trained dataset in the extended applications, from which robust pre-trained weights for downstream tasks are obtained, achieving accurate results with only 5% training samples in the LandCover.ai dataset for water-body extraction. To summarize, our method enables the automatic quality assessment of annotated sample data and the construction of high-quality sample data, both of which have the potential to improve the performance of machine learning models in the remote sensing community. The source code and the two constructed AQS datasets will be publicly available at https://github.com/zhilyzhang/AQSNet.