An Improved Application Level Multi-path Streaming Algorithm for HBDP Networks

Data intensive computing applications which require access to geographically distributed data are increasingly important in the grid and cloud computing field. We designed a novel approach to the task of transferring bulk data between remote hosts, based on the fork-join queueing model, by combining, at application level, the use of classical pipes as a mean of serializing data locally with multiple parallel TCP connections which ensure high bandwidth throughput. Our new experiments in 1 Gbps and up to 9 Gbps high bandwidth networks have proven the validity of our improved multi-path transmission algorithms. Both simple file transfers and arbitrary data streaming were done effectively, efficiently, in parallel, between pairs of distributed processes connected via extended fast pipes.