Stick to Your Role! Context-dependence and Stability of Personal Value Expression in Large Language Models
arxiv(2024)
摘要
The standard way to study Large Language Models (LLMs) with benchmarks or
psychology questionnaires is to provide many different queries from similar
minimal contexts (e.g. multiple choice questions). However, due to LLMs' highly
context-dependent nature, conclusions from such minimal-context evaluations may
be little informative about the model's behavior in deployment (where it will
be exposed to many new contexts). We argue that context-dependence
(specifically, value stability) should be studied a specific property of LLMs
and used as another dimension of LLM comparison (alongside others such as
cognitive abilities, knowledge, or model size). We present a case-study on the
stability of value expression over different contexts (simulated conversations
on different topics) as measured using a standard psychology questionnaire
(PVQ) and on behavioral downstream tasks. Reusing methods from psychology, we
study Rank-order stability on the population (interpersonal) level, and
Ipsative stability on the individual (intrapersonal) level. We consider two
settings (with and without instructing LLMs to simulate particular personas),
two simulated populations, and three downstream tasks. We observe consistent
trends in the stability of models and model families - Mixtral, Mistral,
GPT-3.5 and Qwen families are more stable than LLaMa-2 and Phi. The consistency
of these trends implies that some models exhibit higher value-stability than
others, and that value stability can be estimated with the set of introduced
methodological tools. When instructed to simulate particular personas, LLMs
exhibit low Rank-Order stability, which further diminishes with conversation
length. This highlights the need for future research on LLMs that coherently
simulate different personas. This paper provides a foundational step in that
direction, and, to our knowledge, it is the first study of value stability in
LLMs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要