# Efficient Probabilistic Logic Reasoning with Graph Neural Networks

ICLR, 2020.

EI

Keywords:

probabilistic logic reasoning Markov Logic Networks graph neural networks

Weibo:

Abstract:

Markov Logic Networks (MLNs), which elegantly combine logic rules and probabilistic graphical models, can be used to address many knowledge graph problems. However, inference in MLN is computationally intensive, making the industrial-scale application of MLN very difficult. In recent years, graph neural networks (GNNs) have emerged as eff...More

Introduction

- Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval.
- Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs (Richardson & Domingos, 2006).
- The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data, while the graphical model formalism provides a principled framework for dealing with uncertainty in data.
- Logic rules can only cover a small part of the possible combinations of knowledge graph relations, limiting the application of models that are purely based on logic rules

Highlights

- Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval
- Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs (Richardson & Domingos, 2006)
- We compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10
- This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning
- ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework
- ExpressGNN employs Graph neural networks to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae

Methods

- The authors compare the method with several strong MLN inference algorithms, including MCMC (Gibbs Sampling; Gilks et al (1995); Richardson & Domingos (2006)), Belief Propagation (BP; Yedidia et al (2001)), Lifted Belief Propagation (Lifted BP; Singla & Domingos (2008)), MC-SAT (Poon & Domingos, 2006) and Hinge-Loss Markov Random Field (HL-MRF; Bach et al (2015); Srinivasan et al (2019)).

Inference accuracy. - Since none of the aforementioned MLN inference methods can scale up to this dataset, the authors compare ExpressGNN with a number of state-of-the-art methods for knowledge base completion, including Neural Tensor Network (NTN; Socher et al (2013)), Neural LP (Yang et al, 2017), DistMult (Kadlec et al, 2017), ComplEx (Trouillon et al, 2016), TransE (Bordes et al, 2013), RotatE (Sun et al, 2019) and pLogicNet (Qu & Tang, 2019).
- The experimental results on the full training data are reported in Table 3 (100% columns)
- Both ExpressGNN-E and ExpressGNN-EM significantly outperform all the baseline methods.
- Compared to knowledge graph embedding methods such as TransE and RotatE, ExpressGNN can leverage the prior knowledge in logic rules and outperform these purely data-driven methods

Results

- Following existing studies (Richardson & Domingos, 2006; Singla & Domingos, 2005), the authors use area under the precision-recall curve (AUC-PR) to evaluate the inference accuracy.
- To evaluate the inference efficiency, the authors use wall-clock running time in minutes.
- Following existing studies (Bordes et al, 2013; Sun et al, 2019), the authors use filtered ranking where the test triples are ranked against all the candidate triples not appearing in the dataset.
- The authors compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10

Conclusion

- This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning.
- ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework.
- ExpressGNN employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae.
- ExpressGNN is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN and the embedding part

Summary

## Introduction:

Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval.- Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs (Richardson & Domingos, 2006).
- The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data, while the graphical model formalism provides a principled framework for dealing with uncertainty in data.
- Logic rules can only cover a small part of the possible combinations of knowledge graph relations, limiting the application of models that are purely based on logic rules
## Methods:

The authors compare the method with several strong MLN inference algorithms, including MCMC (Gibbs Sampling; Gilks et al (1995); Richardson & Domingos (2006)), Belief Propagation (BP; Yedidia et al (2001)), Lifted Belief Propagation (Lifted BP; Singla & Domingos (2008)), MC-SAT (Poon & Domingos, 2006) and Hinge-Loss Markov Random Field (HL-MRF; Bach et al (2015); Srinivasan et al (2019)).

Inference accuracy.- Since none of the aforementioned MLN inference methods can scale up to this dataset, the authors compare ExpressGNN with a number of state-of-the-art methods for knowledge base completion, including Neural Tensor Network (NTN; Socher et al (2013)), Neural LP (Yang et al, 2017), DistMult (Kadlec et al, 2017), ComplEx (Trouillon et al, 2016), TransE (Bordes et al, 2013), RotatE (Sun et al, 2019) and pLogicNet (Qu & Tang, 2019).
- The experimental results on the full training data are reported in Table 3 (100% columns)
- Both ExpressGNN-E and ExpressGNN-EM significantly outperform all the baseline methods.
- Compared to knowledge graph embedding methods such as TransE and RotatE, ExpressGNN can leverage the prior knowledge in logic rules and outperform these purely data-driven methods
## Results:

Following existing studies (Richardson & Domingos, 2006; Singla & Domingos, 2005), the authors use area under the precision-recall curve (AUC-PR) to evaluate the inference accuracy.- To evaluate the inference efficiency, the authors use wall-clock running time in minutes.
- Following existing studies (Bordes et al, 2013; Sun et al, 2019), the authors use filtered ranking where the test triples are ranked against all the candidate triples not appearing in the dataset.
- The authors compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10
## Conclusion:

This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning.- ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework.
- ExpressGNN employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae.
- ExpressGNN is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN and the embedding part

- Table1: Inference accuracy (AUC-PR) of different methods on three benchmark datasets
- Table2: AUC-PR for different combiand expressiveness of model by tuning the dimensionality of nations of GNN and tunable embeddings
- Table3: Performance on FB15K-237 with varied training set size
- Table4: Zero-shot learning performance on FB15K-237
- Table5: Complete statistics of the benchmark datasets
- Table6: Inference performance of competitors and our method under the closed-world semantics
- Table7: Examples of logic formulae used in four benchmark datasets

Related work

- Statistical relational learning. There is an extensive literature relating the topic of logic reasoning. Here we only focus on the approaches that are most relevant to statistical relational learning on knowledge graphs. Logic rules can compactly encode the domain knowledge and complex dependencies. Thus, hard logic rules are widely used for reasoning in earlier attempts, such as expert systems (Ignizio, 1991) and inductive logic programming (Muggleton & De Raedt, 1994). However, hard logic is very brittle and has difficulty in coping with uncertainty in both the logic rules and the facts in knowledge graphs. Later studies have explored to introduce probabilistic graphical model in logic reasoning, seeking to combine the advantages of relational and probabilistic approaches. Representative works including Relational Markov Networks (RMNs; Taskar et al (2007)) and Markov Logic Networks (MLNs; Richardson & Domingos (2006)) were proposed in this background.

Funding

- We acknowledge grants from NSF IIS-1218749, NIH BIGDATA 1R01GM108341, NSF CAREER IIS-1350983, NSF IIS-1639792 EAGER, NSF IIS-1841351 EA-GER, NSF CNS-1704701, ONR N00014-15-1-2340, Intel ISTC, Nvidia, Google, Amazon AWS and Siemens
- Yuyu Zhang is supported by the Siemens FutureMaker Fellowship

Reference

- Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. arXiv preprint arXiv:1505.04406, 2015.
- Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
- Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247–1250. AcM, 2008.
- Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pp. 2787–2795, 2013.
- Hanjun Dai, Bo Dai, and Le Song. Discriminative embeddings of latent variable models for structured data. In International conference on machine learning, pp. 2702–2711, 2016.
- Woodrow W Denham. The detection of patterns in Alyawara nonverbal behavior. PhD thesis, University of Washington, Seattle., 1973.
- Zoubin Ghahramani, Matthew J Beal, et al. Graphical models and variational methods. Advanced mean field methods-theory and practice. MIT Press, 2000.
- Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. Markov chain Monte Carlo in practice. Chapman and Hall/CRC, 1995.
- Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303–1347, 2013.
- James Ignizio. Introduction to expert systems, volume 21. 1991.
- Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744, 2017.
- Tushar Khot, Sriraam Natarajan, Kristian Kersting, and Jude Shavlik. Learning markov logic networks via functional gradient boosting. In 2011 IEEE 11th International Conference on Data Mining, pp. 320–329. IEEE, 2011.
- Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
- Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In ICLR, 2016.
- Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. arXiv preprint arXiv:1905.11136, 2019.
- Lilyana Mihalkova and Raymond J Mooney. Bottom-up learning of markov logic network structure. In Proceedings of the 24th international conference on Machine learning, pp. 625–632. ACM, 2007.
- Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19:629–679, 1994.
- Hoifung Poon and Pedro Domingos. Sound and efficient inference with probabilistic and deterministic dependencies. In AAAI, volume 6, pp. 458–463, 2006.
- Hoifung Poon and Pedro Domingos. Joint inference in information extraction. In AAAI, volume 7, pp. 913–918, 2007.
- Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. arXiv preprint arXiv:1906.08495, 2019.
- Meng Qu, Yoshua Bengio, and Jian Tang. GMNN: Graph Markov neural networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 5241–5250, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
- Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62(1-2): 107–136, 2006.
- Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(Sep): 2539–2561, 2011.
- Parag Singla and Pedro Domingos. Discriminative training of markov logic networks. In AAAI, volume 5, pp. 868–873, 2005.
- Parag Singla and Pedro Domingos. Entity resolution with markov logic. In Data Mining, 2006. ICDM’06. Sixth International Conference on, pp. 572–582. IEEE, 2006a.
- Parag Singla and Pedro Domingos. Memory-efficient inference in relational domains. In AAAI, volume 6, pp. 488–493, 2006b.
- Parag Singla and Pedro M Domingos. Lifted first-order belief propagation. In AAAI, volume 8, pp. 1094–1099, 2008.
- Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pp. 926–934, 2013.
- Sriram Srinivasan, Behrouz Babaki, Golnoosh Farnadi, and Lise Getoor. Lifted hinge-loss markov random fields. AAAI, 2019.
- Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.
- Ben Taskar, Pieter Abbeel, Ming-Fai Wong, and Daphne Koller. Relational markov networks. Introduction to statistical relational learning, pp. 175–200, 2007.
- Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pp. 57–66, 2015.
- Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071–2080, 2016.
- Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. One-shot relational learning for knowledge graphs. arXiv preprint arXiv:1808.09040, 2018.
- Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
- Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base completion. CoRR, abs/1702.08367, 2017.
- Jonathan S Yedidia, William T Freeman, and Yair Weiss. Generalized belief propagation. In Advances in neural information processing systems, pp. 689–695, 2001.
- Weizhe Zhang, Xiaoqiang Li, Hui He, and Xing Wang. Identifying network public opinion leaders based on markov logic networks. The scientific world journal, 2014, 2014.

Tags

Comments