An Empirical and Conceptual Categorization of Value-based Exploration Methods

This paper categorizes several exploration methods for reinforcement learning according to their underlying heuristics. Using linear Q-learning, we compare representative methods from each category of methods on a set of domains designed to pose different exploratory challenges. We find that the relative performance of each method depends on the specific exploratory challenge posed by the domain. Our results suggest that each exploration heuristic encodes a bias which is appropriate for a subset of environments.