Adaptive Primal-Dual Method for Safe Reinforcement Learning
CoRR(2024)
摘要
Primal-dual methods have a natural application in Safe Reinforcement Learning
(SRL), posed as a constrained policy optimization problem. In practice however,
applying primal-dual methods to SRL is challenging, due to the inter-dependency
of the learning rate (LR) and Lagrangian multipliers (dual variables) each time
an embedded unconstrained RL problem is solved. In this paper, we propose,
analyze and evaluate adaptive primal-dual (APD) methods for SRL, where two
adaptive LRs are adjusted to the Lagrangian multipliers so as to optimize the
policy in each iteration. We theoretically establish the convergence,
optimality and feasibility of the APD algorithm. Finally, we conduct numerical
evaluation of the practical APD algorithm with four well-known environments in
Bullet-Safey-Gym employing two state-of-the-art SRL algorithms: PPO-Lagrangian
and DDPG-Lagrangian. All experiments show that the practical APD algorithm
outperforms (or achieves comparable performance) and attains more stable
training than the constant LR cases. Additionally, we substantiate the
robustness of selecting the two adaptive LRs by empirical evidence.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要