Theoretical Hardness and Tractability of POMDPs in RL with Partial Online State Information
arXiv (Cornell University)(2023)
摘要
Partially observable Markov decision processes (POMDPs) have been widely
applied in various real-world applications. However, existing theoretical
results have shown that learning in POMDPs is intractable in the worst case,
where the main challenge lies in the lack of latent state information. A key
fundamental question here is: how much online state information (OSI) is
sufficient to achieve tractability? In this paper, we establish a lower bound
that reveals a surprising hardness result: unless we have full OSI, we need an
exponentially scaling sample complexity to obtain an ϵ-optimal policy
solution for POMDPs. Nonetheless, inspired by the insights in our lower-bound
design, we identify important tractable subclasses of POMDPs, even with only
partial OSI. In particular, for two subclasses of POMDPs with partial OSI, we
provide new algorithms that are proved to be near-optimal by establishing new
regret upper and lower bounds. Both our algorithm design and regret analysis
involve non-trivial developments for joint OSI query and action control.
更多查看译文
关键词
pomdps,theoretical hardness,tractability,rl
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要