Efficient Probabilistic Logic Reasoning with Graph Neural Networks

ICLR, 2020.

Cited by: 6|Bibtex|Views220|Links
EI
Keywords:
probabilistic logic reasoning Markov Logic Networks graph neural networks
Weibo:
This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning

Abstract:

Markov Logic Networks (MLNs), which elegantly combine logic rules and probabilistic graphical models, can be used to address many knowledge graph problems. However, inference in MLN is computationally intensive, making the industrial-scale application of MLN very difficult. In recent years, graph neural networks (GNNs) have emerged as eff...More
0
Introduction
  • Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval.
  • Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs (Richardson & Domingos, 2006).
  • The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data, while the graphical model formalism provides a principled framework for dealing with uncertainty in data.
  • Logic rules can only cover a small part of the possible combinations of knowledge graph relations, limiting the application of models that are purely based on logic rules
Highlights
  • Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval
  • Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs (Richardson & Domingos, 2006)
  • We compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10
  • This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning
  • ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework
  • ExpressGNN employs Graph neural networks to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae
Methods
  • The authors compare the method with several strong MLN inference algorithms, including MCMC (Gibbs Sampling; Gilks et al (1995); Richardson & Domingos (2006)), Belief Propagation (BP; Yedidia et al (2001)), Lifted Belief Propagation (Lifted BP; Singla & Domingos (2008)), MC-SAT (Poon & Domingos, 2006) and Hinge-Loss Markov Random Field (HL-MRF; Bach et al (2015); Srinivasan et al (2019)).

    Inference accuracy.
  • Since none of the aforementioned MLN inference methods can scale up to this dataset, the authors compare ExpressGNN with a number of state-of-the-art methods for knowledge base completion, including Neural Tensor Network (NTN; Socher et al (2013)), Neural LP (Yang et al, 2017), DistMult (Kadlec et al, 2017), ComplEx (Trouillon et al, 2016), TransE (Bordes et al, 2013), RotatE (Sun et al, 2019) and pLogicNet (Qu & Tang, 2019).
  • The experimental results on the full training data are reported in Table 3 (100% columns)
  • Both ExpressGNN-E and ExpressGNN-EM significantly outperform all the baseline methods.
  • Compared to knowledge graph embedding methods such as TransE and RotatE, ExpressGNN can leverage the prior knowledge in logic rules and outperform these purely data-driven methods
Results
  • Following existing studies (Richardson & Domingos, 2006; Singla & Domingos, 2005), the authors use area under the precision-recall curve (AUC-PR) to evaluate the inference accuracy.
  • To evaluate the inference efficiency, the authors use wall-clock running time in minutes.
  • Following existing studies (Bordes et al, 2013; Sun et al, 2019), the authors use filtered ranking where the test triples are ranked against all the candidate triples not appearing in the dataset.
  • The authors compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10
Conclusion
  • This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning.
  • ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework.
  • ExpressGNN employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae.
  • ExpressGNN is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN and the embedding part
Summary
  • Introduction:

    Knowledge graphs collect and organize relations and attributes about entities, which are playing an increasingly important role in many applications, including question answering and information retrieval.
  • Markov Logic Networks (MLNs) were proposed to combine hard logic rules and probabilistic graphical models, which can be applied to various tasks on knowledge graphs (Richardson & Domingos, 2006).
  • The logic rules incorporate prior knowledge and allow MLNs to generalize in tasks with small amount of labeled data, while the graphical model formalism provides a principled framework for dealing with uncertainty in data.
  • Logic rules can only cover a small part of the possible combinations of knowledge graph relations, limiting the application of models that are purely based on logic rules
  • Methods:

    The authors compare the method with several strong MLN inference algorithms, including MCMC (Gibbs Sampling; Gilks et al (1995); Richardson & Domingos (2006)), Belief Propagation (BP; Yedidia et al (2001)), Lifted Belief Propagation (Lifted BP; Singla & Domingos (2008)), MC-SAT (Poon & Domingos, 2006) and Hinge-Loss Markov Random Field (HL-MRF; Bach et al (2015); Srinivasan et al (2019)).

    Inference accuracy.
  • Since none of the aforementioned MLN inference methods can scale up to this dataset, the authors compare ExpressGNN with a number of state-of-the-art methods for knowledge base completion, including Neural Tensor Network (NTN; Socher et al (2013)), Neural LP (Yang et al, 2017), DistMult (Kadlec et al, 2017), ComplEx (Trouillon et al, 2016), TransE (Bordes et al, 2013), RotatE (Sun et al, 2019) and pLogicNet (Qu & Tang, 2019).
  • The experimental results on the full training data are reported in Table 3 (100% columns)
  • Both ExpressGNN-E and ExpressGNN-EM significantly outperform all the baseline methods.
  • Compared to knowledge graph embedding methods such as TransE and RotatE, ExpressGNN can leverage the prior knowledge in logic rules and outperform these purely data-driven methods
  • Results:

    Following existing studies (Richardson & Domingos, 2006; Singla & Domingos, 2005), the authors use area under the precision-recall curve (AUC-PR) to evaluate the inference accuracy.
  • To evaluate the inference efficiency, the authors use wall-clock running time in minutes.
  • Following existing studies (Bordes et al, 2013; Sun et al, 2019), the authors use filtered ranking where the test triples are ranked against all the candidate triples not appearing in the dataset.
  • The authors compute the Mean Reciprocal Ranks (MRR), which is the average of the reciprocal rank of all the truth queries, and Hits@10, which is the percentage of truth queries that are ranked among the top 10
  • Conclusion:

    This paper studies the probabilistic logic reasoning problem, and proposes ExpressGNN to combine the advantages of Markov Logic Networks in logic reasoning and graph neural networks in graph representation learning.
  • ExpressGNN addresses the scalability issue of Markov Logic Networks with efficient stochastic training in the variational EM framework.
  • ExpressGNN employs GNNs to capture the structure knowledge that is implicitly encoded in the knowledge graph, which serves as supplement to the knowledge from logic formulae.
  • ExpressGNN is a general framework that can trade-off the model compactness and expressiveness by tuning the dimensionality of the GNN and the embedding part
Tables
  • Table1: Inference accuracy (AUC-PR) of different methods on three benchmark datasets
  • Table2: AUC-PR for different combiand expressiveness of model by tuning the dimensionality of nations of GNN and tunable embeddings
  • Table3: Performance on FB15K-237 with varied training set size
  • Table4: Zero-shot learning performance on FB15K-237
  • Table5: Complete statistics of the benchmark datasets
  • Table6: Inference performance of competitors and our method under the closed-world semantics
  • Table7: Examples of logic formulae used in four benchmark datasets
Download tables as Excel
Related work
  • Statistical relational learning. There is an extensive literature relating the topic of logic reasoning. Here we only focus on the approaches that are most relevant to statistical relational learning on knowledge graphs. Logic rules can compactly encode the domain knowledge and complex dependencies. Thus, hard logic rules are widely used for reasoning in earlier attempts, such as expert systems (Ignizio, 1991) and inductive logic programming (Muggleton & De Raedt, 1994). However, hard logic is very brittle and has difficulty in coping with uncertainty in both the logic rules and the facts in knowledge graphs. Later studies have explored to introduce probabilistic graphical model in logic reasoning, seeking to combine the advantages of relational and probabilistic approaches. Representative works including Relational Markov Networks (RMNs; Taskar et al (2007)) and Markov Logic Networks (MLNs; Richardson & Domingos (2006)) were proposed in this background.
Funding
  • We acknowledge grants from NSF IIS-1218749, NIH BIGDATA 1R01GM108341, NSF CAREER IIS-1350983, NSF IIS-1639792 EAGER, NSF IIS-1841351 EA-GER, NSF CNS-1704701, ONR N00014-15-1-2340, Intel ISTC, Nvidia, Google, Amazon AWS and Siemens
  • Yuyu Zhang is supported by the Siemens FutureMaker Fellowship
Reference
  • Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. Hinge-loss markov random fields and probabilistic soft logic. arXiv preprint arXiv:1505.04406, 2015.
    Findings
  • Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
    Findings
  • Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp. 1247–1250. AcM, 2008.
    Google ScholarLocate open access versionFindings
  • Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, pp. 2787–2795, 2013.
    Google ScholarLocate open access versionFindings
  • Hanjun Dai, Bo Dai, and Le Song. Discriminative embeddings of latent variable models for structured data. In International conference on machine learning, pp. 2702–2711, 2016.
    Google ScholarLocate open access versionFindings
  • Woodrow W Denham. The detection of patterns in Alyawara nonverbal behavior. PhD thesis, University of Washington, Seattle., 1973.
    Google ScholarFindings
  • Zoubin Ghahramani, Matthew J Beal, et al. Graphical models and variational methods. Advanced mean field methods-theory and practice. MIT Press, 2000.
    Google ScholarFindings
  • Walter R Gilks, Sylvia Richardson, and David Spiegelhalter. Markov chain Monte Carlo in practice. Chapman and Hall/CRC, 1995.
    Google ScholarLocate open access versionFindings
  • Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303–1347, 2013.
    Google ScholarLocate open access versionFindings
  • James Ignizio. Introduction to expert systems, volume 21. 1991.
    Google ScholarFindings
  • Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. Knowledge base completion: Baselines strike back. arXiv preprint arXiv:1705.10744, 2017.
    Findings
  • Tushar Khot, Sriraam Natarajan, Kristian Kersting, and Jude Shavlik. Learning markov logic networks via functional gradient boosting. In 2011 IEEE 11th International Conference on Data Mining, pp. 320–329. IEEE, 2011.
    Google ScholarLocate open access versionFindings
  • Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
    Findings
  • Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
    Google ScholarLocate open access versionFindings
  • Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In ICLR, 2016.
    Google ScholarLocate open access versionFindings
  • Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. arXiv preprint arXiv:1905.11136, 2019.
    Findings
  • Lilyana Mihalkova and Raymond J Mooney. Bottom-up learning of markov logic network structure. In Proceedings of the 24th international conference on Machine learning, pp. 625–632. ACM, 2007.
    Google ScholarLocate open access versionFindings
  • Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19:629–679, 1994.
    Google ScholarLocate open access versionFindings
  • Hoifung Poon and Pedro Domingos. Sound and efficient inference with probabilistic and deterministic dependencies. In AAAI, volume 6, pp. 458–463, 2006.
    Google ScholarLocate open access versionFindings
  • Hoifung Poon and Pedro Domingos. Joint inference in information extraction. In AAAI, volume 7, pp. 913–918, 2007.
    Google ScholarLocate open access versionFindings
  • Meng Qu and Jian Tang. Probabilistic logic neural networks for reasoning. arXiv preprint arXiv:1906.08495, 2019.
    Findings
  • Meng Qu, Yoshua Bengio, and Jian Tang. GMNN: Graph Markov neural networks. In Proceedings of the 36th International Conference on Machine Learning, pp. 5241–5250, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
    Google ScholarLocate open access versionFindings
  • Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62(1-2): 107–136, 2006.
    Google ScholarLocate open access versionFindings
  • Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(Sep): 2539–2561, 2011.
    Google ScholarLocate open access versionFindings
  • Parag Singla and Pedro Domingos. Discriminative training of markov logic networks. In AAAI, volume 5, pp. 868–873, 2005.
    Google ScholarLocate open access versionFindings
  • Parag Singla and Pedro Domingos. Entity resolution with markov logic. In Data Mining, 2006. ICDM’06. Sixth International Conference on, pp. 572–582. IEEE, 2006a.
    Google ScholarLocate open access versionFindings
  • Parag Singla and Pedro Domingos. Memory-efficient inference in relational domains. In AAAI, volume 6, pp. 488–493, 2006b.
    Google ScholarLocate open access versionFindings
  • Parag Singla and Pedro M Domingos. Lifted first-order belief propagation. In AAAI, volume 8, pp. 1094–1099, 2008.
    Google ScholarLocate open access versionFindings
  • Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pp. 926–934, 2013.
    Google ScholarLocate open access versionFindings
  • Sriram Srinivasan, Behrouz Babaki, Golnoosh Farnadi, and Lise Getoor. Lifted hinge-loss markov random fields. AAAI, 2019.
    Google ScholarLocate open access versionFindings
  • Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019.
    Findings
  • Ben Taskar, Pieter Abbeel, Ming-Fai Wong, and Daphne Koller. Relational markov networks. Introduction to statistical relational learning, pp. 175–200, 2007.
    Google ScholarFindings
  • Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, pp. 57–66, 2015.
    Google ScholarLocate open access versionFindings
  • Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071–2080, 2016.
    Google ScholarLocate open access versionFindings
  • Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. One-shot relational learning for knowledge graphs. arXiv preprint arXiv:1808.09040, 2018.
    Findings
  • Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
    Findings
  • Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base completion. CoRR, abs/1702.08367, 2017.
    Findings
  • Jonathan S Yedidia, William T Freeman, and Yair Weiss. Generalized belief propagation. In Advances in neural information processing systems, pp. 689–695, 2001.
    Google ScholarLocate open access versionFindings
  • Weizhe Zhang, Xiaoqiang Li, Hui He, and Xing Wang. Identifying network public opinion leaders based on markov logic networks. The scientific world journal, 2014, 2014.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments