A Performance Evaluation of Spark GraphFrames for Fast and Scalable Graph Analytics at Twitter.

Graph analytics demand is emerging rapidly and has become one of the key parts of Twitter machine learning for driving engagement, serving most relevant content, and promoting healthier conversations. However, due to lack of infrastructure support for graph analytics, we are suffering from a long timeline and huge engineering effort for each project to deal with graphs at the Twitter scale, which blocks us from fast iteration. To bring fast and scalable graph analytics capability into Twitter, we adopted Spark GraphFrames and conducted a performance evaluation for a typical graph analytics use case - Connected Component - on one of the largest graphs at Twitter. Compared to our existing Scalding solution, Spark GraphFrames achieved more than 33x speedup in running time.