Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs
CoRR(2024)
摘要
Due to the widespread use of large language models (LLMs) in ubiquitous
systems, we need to understand whether they embed a specific worldview and what
these views reflect. Recent studies report that, prompted with political
questionnaires, LLMs show left-liberal leanings. However, it is as yet unclear
whether these leanings are reliable (robust to prompt variations) and whether
the leaning is consistent across policies and political leaning. We propose a
series of tests which assess the reliability and consistency of LLMs' stances
on political statements based on a dataset of voting-advice questionnaires
collected from seven EU countries and annotated for policy domains. We study
LLMs ranging in size from 7B to 70B parameters and find that their reliability
increases with parameter count. Larger models show overall stronger alignment
with left-leaning parties but differ among policy programs: They evince a
(left-wing) positive stance towards environment protection, social welfare but
also (right-wing) law and order, with no consistent preferences in foreign
policy, migration, and economy.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要