AI helps you reading Science
AI generates interpretation videos
AI extracts and analyses the key points of the paper to generate videos automatically
AI parses the academic lineage of this thesis
AI extracts a summary of this paper
We proposed PinSage, a random-walk graph convolutional network
Graph Convolutional Neural Networks for Web-Scale Recommender Systems.
KDD, (2018): 974-983
Recent advancements in deep neural networks for graph-structured data have led to state-of-the-art performance on recommender system benchmarks. However, making these methods practical and scalable to web-scale recommendation tasks with billions of items and hundreds of millions of users remains an unsolved challenge. Here we describe a l...More
PPT (Upload PPT)
- Deep learning methods have an increasingly critical role in recommender system applications, being used to learn useful lowdimensional embeddings of images, text, and even individual users [9, 12].
- Recent years have seen significant developments in this space— especially the development of new deep learning methods that are capable of learning on graph-structured data, which is fundamental for recommendation applications [6, 19, 21, 24, 29, 30].
- These gains on benchmark tasks have yet to be translated to gains in real-world production environments
- Deep learning methods have an increasingly critical role in recommender system applications, being used to learn useful lowdimensional embeddings of images, text, and even individual users [9, 12]
- Most prominent among these recent advancements is the success of deep learning architectures known as Graph Convolutional Networks (GCNs) [19, 21, 24, 29]
- Our work demonstrates the substantial impact that Graph Convolutional Network have on recommendation performance in a real-world environment
- We evaluate the methods using Mean Reciprocal Rank (MRR), which takes into account of the rank of the item j among recommended items for query item q: Mean Reciprocal Rank = n (q,i)∈ L
- We proposed PinSage, a random-walk graph convolutional network (GCN)
- Our work demonstrates the impact that graph convolutional methods can have in a production recommender system, and we believe that PinSage can be further extended in the future to tackle other graph representation learning problems at large scale, including knowledge graph reasoning and graph clustering
- The authors describe the technical details of the PinSage architecture and training, as well as a MapReduce pipeline to efficiently generate embeddings using a trained PinSage model.
The key computational workhorse of the approach is the notion of localized graph convolutions.1 To generate the embedding for a node, the authors apply multiple convolutional modules that aggregate feature information from the node’s local graph neighborhood (Figure 1).
- Each module learns how to aggregate information from a small graph neighborhood, and by stacking multiple such modules, the approach can gain information about the local network topology.
- The authors define the hit-rate as the fraction of queries q where i was ranked among the top K of the test sample
- This metric directly measures the probability that recommendations made by the algorithm contain the items related to the query pin q.
- Through extensive offline metrics, controlled user studies, and A/B tests, the authors show that the approach achieves state-of-the-art performance compared to other scalable deep content-based recommendation algorithms, in both an item-item recommendation task, as well as a “homefeed” recommendation task.
- In offline ranking metrics the authors improve over the best performing baseline by more than 40%, in head-to-head human evaluations the recommendations are preferred about 60% of the time, and the A/B tests show 30% to 100% improvements in user engagement across various settings.
- The authors introduce a number of new training techniques to improve performance and a MapReduce inference pipeline to scale up to graphs with billions of nodes
- PinSage is a highly-scalable GCN algorithm capable of learning embeddings for nodes in web-scale graphs containing billions of objects.
- The authors deployed PinSage at Pinterest and comprehensively evaluated the quality of the learned embeddings on a number of recommendation tasks, with offline metrics, user studies and A/B tests all demonstrating a substantial improvement in recommendation performance.
- The authors' work demonstrates the impact that graph convolutional methods can have in a production recommender system, and the authors believe that PinSage can be further extended in the future to tackle other graph representation learning problems at large scale, including knowledge graph reasoning and graph clustering
- Table1: Hit-rate and MRR for PinSage and content-based deep learning baselines. Overall, PinSage gives 150% improvement in hit rate and 60% improvement in MRR over the best baseline.5
- Table2: Head-to-head comparison of which image is more relevant to the recommended query image
- Table3: Runtime comparisons for different batch sizes
- Table4: Performance tradeoffs for importance pooling
- Our work builds upon a number of recent advancements in deep learning methods for graph-structured data.
The notion of neural networks for graph data was first outlined in Gori et al (2005)  and further elaborated on in Scarselli et al (2009) . However, these initial approaches to deep learning on graphs required running expensive neural “message-passing” algorithms to convergence and were prohibitively expensive on large graphs. Some limitations were addressed by Gated Graph Sequence Neural Networks —which employs modern recurrent neural architectures—but the approach remains computationally expensive and has mainly been used on graphs with <10, 000 nodes.
More recently, there has been a surge of methods that rely on the notion of “graph convolutions” or Graph Convolutional Networks (GCNs). This approach originated with the work of Bruna et al (2013), which developed a version of graph convolutions based on spectral graph thery . Following on this work, a number of authors proposed improvements, extensions, and approximations of these spectral convolutions [6, 10, 11, 13, 18, 21, 24, 29, 31], leading to new state-of-the-art results on benchmarks such as node classification, link prediction, as well as recommender system tasks (e.g., the MovieLens benchmark ). These approaches have consistently outperformed techniques based upon matrix factorization or random walks (e.g., node2vec  and DeepWalk ), and their success has led to a surge of interest in applying GCN-based methods to applications ranging from recommender systems  to drug design [20, 31]. Hamilton et al (2017b)  and Bronstein et al (2017)  provide comprehensive surveys of recent advancements.
- Through extensive offline metrics, controlled user studies, and A/B tests, we show that our approach achieves state-of-the-art performance compared to other scalable deep content-based recommendation algorithms, in both an item-item recommendation task (i.e., related-pin recommendation), as well as a “homefeed” recommendation task
- In offline ranking metrics we improve over the best performing baseline by more than 40%, in head-to-head human evaluations our recommendations are preferred about 60% of the time, and the A/B tests show 30% to 100% improvements in user engagement across various settings
- We also introduce a number of new training techniques to improve performance and a MapReduce inference pipeline to scale up to graphs with billions of nodes
Study subjects and analysis
We use all other pins as negative items (and sample them as described in Section 3.3). Overall, we use 1.2 billion pairs of positive training examples (in addition to 500 negative examples per batch and 6 hard negative examples per pin). Thus in total we use 7.5 billion training examples
- M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).
- A. Andoni and P. Indyk. 2006. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. In FOCS.
- T. Bansal, D. Belanger, and A. McCallum. 2016. Ask the GRU: Multi-task learning for deep text recommendations. In RecSys. ACM.
- Y. Bengio, J. Louradour, R. Collobert, and J. Weston. 2009. Curriculum learning. In ICML.
- A. Z. Broder, D. Carmel, M. Herscovici, A. Soffer, and J. Zien. 2003. Efficient query evaluation using a two-level retrieval process. In CIKM.
- M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. 2017. Geometric deep learning: Going beyond euclidean data. IEEE Signal Processing Magazine 34, 4 (2017).
- J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. 2014. Spectral networks and locally connected networks on graphs. In ICLR.
- J. Chen, T. Ma, and C. Xiao. 201FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling. ICLR (2018).
- P. Covington, J. Adams, and E. Sargin. 2016. Deep neural networks for youtube recommendations. In RecSys. ACM.
- H. Dai, B. Dai, and L. Song. 2016. Discriminative Embeddings of Latent Variable Models for Structured Data. In ICML.
- M. Defferrard, X. Bresson, and P. Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS.
- A. Van den Oord, S. Dieleman, and B. Schrauwen. 2013. Deep content-based music recommendation. In NIPS.
- D. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In NIPS.
- C. Eksombatchai, P. Jindal, J. Z. Liu, Y. Liu, R. Sharma, C. Sugnet, M. Ulrich, and J. Leskovec. 2018. Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time. WWW (2018).
- M. Gori, G. Monfardini, and F. Scarselli. 2005. A new model for learning in graph domains. In IEEE International Joint Conference on Neural Networks.
- P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677 (2017).
- A. Grover and J. Leskovec. 2016. node2vec: Scalable feature learning for networks. In KDD.
- W. L. Hamilton, R. Ying, and J. Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NIPS.
- W. L. Hamilton, R. Ying, and J. Leskovec. 2017. Representation Learning on Graphs: Methods and Applications. IEEE Data Engineering Bulletin (2017).
- S. Kearnes, K. McCloskey, M. Berndl, V. Pande, and P. Riley. 2016. Molecular graph convolutions: moving beyond fingerprints. CAMD 30, 8.
- T. N. Kipf and M. Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR.
- Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. 2015. Gated graph sequence neural networks. In ICLR.
- T. Mikolov, I Sutskever, K. Chen, G. S. Corrado, and J. Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
- F. Monti, M. M. Bronstein, and X. Bresson. 2017. Geometric matrix completion with recurrent multi-graph neural networks. In NIPS.
- OpenMP Architecture Review Board. 2015. OpenMP Application Program Interface Version 4.5. (2015).
- B. Perozzi, R. Al-Rfou, and S. Skiena. 2014. DeepWalk: Online learning of social representations. In KDD.
- F. Scarselli, M. Gori, A.C. Tsoi, M. Hagenbuchner, and G. Monfardini. 2009. The graph neural network model. IEEE Transactions on Neural Networks 20, 1 (2009), 61–80.
- K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
- R. van den Berg, T. N. Kipf, and M. Welling. 2017. Graph Convolutional Matrix Completion. arXiv preprint arXiv:1706.02263 (2017).
- J. You, R. Ying, X. Ren, W. L. Hamilton, and J. Leskovec. 2018. GraphRNN: Generating Realistic Graphs using Deep Auto-regressive Models. ICML (2018).
- M. Zitnik, M. Agrawal, and J. Leskovec. 2018. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics (2018).