KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge
CoRR(2024)
摘要
For Large Language Models (LLMs) to be effectively deployed in a specific
country, they must possess an understanding of the nation's culture and basic
knowledge. To this end, we introduce National Alignment, which measures an
alignment between an LLM and a targeted country from two aspects: social value
alignment and common knowledge alignment. Social value alignment evaluates how
well the model understands nation-specific social values, while common
knowledge alignment examines how well the model captures basic knowledge
related to the nation. We constructed KorNAT, the first benchmark that measures
national alignment with South Korea. For the social value dataset, we obtained
ground truth labels from a large-scale survey involving 6,174 unique Korean
participants. For the common knowledge dataset, we constructed samples based on
Korean textbooks and GED reference materials. KorNAT contains 4K and 6K
multiple-choice questions for social value and common knowledge, respectively.
Our dataset creation process is meticulously designed and based on statistical
sampling theory and was refined through multiple rounds of human review. The
experiment results of seven LLMs reveal that only a few models met our
reference score, indicating a potential for further enhancement. KorNAT has
received government approval after passing an assessment conducted by a
government-affiliated organization dedicated to evaluating dataset quality.
Samples and detailed evaluation protocols of our dataset can be found in
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要