When One Pipeline Is Not Enough

Pipelines that operate on buffers often work well to mitigat e the high latency inherent in interprocessor communication and in accessing data on disk. Running a single pipe line on each node works well when each pipeline stage consumes and produces data at the same rate. If a stage might c onsume data faster or slower than it produces data, a single pipeline becomes unwieldy. We describe how we have extended the FG programming environm ent to support multiple pipelines in two forms. When a node might send and receive data at different rates dur ing interprocessor communication, we use disjoint pipelines that send and receive on each node. When a node cons umes and produces data from different streams on the node, we use multiple pipelines that intersect at a particul ar stage. Experimental results for two out-of-core sorting algorithms—one based on columnsort and the other a distribu tion-based sort—demonstrate the value of multiple pipelines. ∗Work performed while Elena Riccio Davidson was at Dartmouth College.