Online Performance Management Using Hybrid Reinforcement Learning

We present a new hybrid approach to performance man- agement, combining disparate strengths of Reinforcment Learning (RL) with model-based (e.g. queuing-theoretic) ap- proaches. Our method trains nonlinear function approxima- tors using offline RL on data collected while a model-based policy controls the system. By training offline we avoid po- tentially poor performance in live online training, while f unc- tion approximation allows generalization across both states and actions, so that the need for exploratory actions may be greatly reduced. Our results show that, in a prototype re- source allocation scenario among multiple web applications, hybrid RL training can achieve significant performance im- provements over a variety of initial queuing model-based policies. We also find that, as expected, RL can deal effec- tively with both transients and switching delays, which lie outside the scope of traditional steady-state queuing theo ry.