WeChat Mini Program
Old Version Features

    • Integration of Multi-Source Landslide Disaster Data Based on Flink Framework and APSO Load Balancing Task Scheduling

    Zongmin Wang, Huangtaojun Liang,Haibo Yang, Mengyu Li,Yingchun Cai

    ISPRS INTERNATIONAL JOURNAL OF GEO-INFORMATION（2025）

    Zhengzhou Univ

    Cited 0|Views0
    Abstract
    As monitoring technologies and data collection methodologies advance, landslide disaster data reflects attributes such as diverse sources, heterogeneity, substantial volumes, and stringent real-time requirements. To bolster the data support capabilities for the monitoring, prevention, and management of landslide disasters, the efficient integration of multi-source heterogeneous data is of paramount importance. The present study proposes an innovative approach to integrate multi-source landslide disaster data by combining the Flink-oriented framework with load balancing task scheduling based on an improved particle swarm optimization (APSO) algorithm. It utilizes Flink’s streaming processing capabilities to efficiently process and store multi-source landslide data. To tackle the issue of uneven cluster load distribution during the integration process, the APSO algorithm is proposed to facilitate cluster load balancing. The findings indicate the following: (1) The multi-source data integration method for landslide disaster based on Flink and APSO proposed in this article, combined with the structural characteristics of landslide disaster data, adopts different integration methods for data in different formats, which can effectively achieve the integration of multi-source landslide data. (2) A multi-source landslide data integration framework based on Flink has been established. Utilizing Kafka as a message queue, a real-time data pipeline was constructed, with Flink facilitating data processing and read/write operations for the database. This implementation achieves efficient integration of multi-source landslide data. (3) Compared to Flink’s default task scheduling strategy, the cluster load balancing strategy based on APSO demonstrated a reduction of approximately 4.7% in average task execution time and an improvement of approximately 5.4% in average system throughput during actual tests using landslide data sets. The research findings illustrate a significant improvement in the efficiency of data integration processing and system performance.
    More
    Translated text
    Key words
    landslide,multi-source data,Flink,data integration,load balancing
    求助PDF
    上传PDF
    View via Publisher
    Bibtex
    AI Read Science
    AI Summary
    AI Summary is the key point extracted automatically understanding the full text of the paper, including the background, methods, results, conclusions, icons and other key content, so that you can get the outline of the paper at a glance.
    Example
    Background
    Key content
    Introduction
    Methods
    Results
    Related work
    Fund
    Key content
    • Pretraining has recently greatly promoted the development of natural language processing (NLP)
    • We show that M6 outperforms the baselines in multimodal downstream tasks, and the large M6 with 10 parameters can reach a better performance
    • We propose a method called M6 that is able to process information of multiple modalities and perform both single-modal and cross-modal understanding and generation
    • The model is scaled to large model with 10 billion parameters with sophisticated deployment, and the 10 -parameter M6-large is the largest pretrained model in Chinese
    • Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities We will continue the pretraining of extremely large models by increasing data to explore the limit of its performance
    Upload PDF to Generate Summary
    Must-Reading Tree
    Example
    Generate MRT to find the research sequence of this paper
    Data Disclaimer
    The page data are from open Internet sources, cooperative publishers and automatic analysis results through AI technology. We do not make any commitments and guarantees for the validity, accuracy, correctness, reliability, completeness and timeliness of the page data. If you have any questions, please contact us by email: report@aminer.cn
    Chat Paper
    Summary is being generated by the instructions you defined