RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization
arxiv(2023)
摘要
Multi-agent systems are characterized by environmental uncertainty, varying
policies of agents, and partial observability, which result in significant
risks. In the context of Multi-Agent Reinforcement Learning (MARL), learning
coordinated and decentralized policies that are sensitive to risk is
challenging. To formulate the coordination requirements in risk-sensitive MARL,
we introduce the Risk-sensitive Individual-Global-Max (RIGM) principle as a
generalization of the Individual-Global-Max (IGM) and Distributional IGM (DIGM)
principles. This principle requires that the collection of risk-sensitive
action selections of each agent should be equivalent to the risk-sensitive
action selection of the central policy. Current MARL value factorization
methods do not satisfy the RIGM principle for common risk metrics such as the
Value at Risk (VaR) metric or distorted risk measurements. Therefore, we
propose RiskQ to address this limitation, which models the joint return
distribution by modeling quantiles of it as weighted quantile mixtures of
per-agent return distribution utilities. RiskQ satisfies the RIGM principle for
the VaR and distorted risk metrics. We show that RiskQ can obtain promising
performance through extensive experiments. The source code of RiskQ is
available in https://github.com/xmu-rl-3dv/RiskQ.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要