KGLens: A Parameterized Knowledge Graph Solution to Assess What an LLM Does and Doesn't Know
CoRR(2023)
Abstract
Measuring the alignment between a Knowledge Graph (KG) and Large Language
Models (LLMs) is an effective method to assess the factualness and identify the
knowledge blind spots of LLMs. However, this approach encounters two primary
challenges including the translation of KGs into natural language and the
efficient evaluation of these extensive and complex structures. In this paper,
we present KGLens–a novel framework aimed at measuring the alignment between
KGs and LLMs, and pinpointing the LLMs' knowledge deficiencies relative to KGs.
KGLens features a graph-guided question generator for converting KGs into
natural language, along with a carefully designed sampling strategy based on
parameterized KG structure to expedite KG traversal. We conducted experiments
using three domain-specific KGs from Wikidata, which comprise over 19,000
edges, 700 relations, and 21,000 entities. Our analysis across eight LLMs
reveals that KGLens not only evaluates the factual accuracy of LLMs more
rapidly but also delivers in-depth analyses on topics, temporal dynamics, and
relationships. Furthermore, human evaluation results indicate that KGLens can
assess LLMs with a level of accuracy nearly equivalent to that of human
annotators, achieving 95.7
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined