Billion node graph inference : iterative processing on The Machine

Iterative graph processing has emerged as a necessary and powerful tool at the heart of solutions to several large scale analytics problems such as malware detection, genome analysis and online advertising. Due to the large amount of communication involved, iterative graph processing on large graphs does not scale well on distributed systems. To demonstrate the benefits of a memory driven computing architecture, we designed and implemented an iterative graph processing engine that uses The Machine’s Fabric-attached-memory as a communication medium. The engine demonstrates near-linear scalability with a 162x speed-up over a state of the art graph processing system on a Superdome X and a projected 85x speed up over the same system on the Machine Fabric Testbed (MFT).