Managing Power Consumption and Performance of Computing Systems Using Reinforcement Learning

Electrical power management in large-scale IT systems such as commercial data- centers is an application area of rapidly growing interest f rom both an economic and ecological perspective, with billions of dollars and mi llions of metric tons of CO2 emissions at stake annually. Businesses want to save power without sac- rificing performance. This paper presents a reinforcement l earning approach to simultaneous online management of both performance and power consumption. We apply RL in a realistic laboratory testbed using a Blade cluster and dynam- ically varying HTTP workload running on a commercial web applications mid- dleware platform. We embed a CPU frequency controller in the Blade servers' firmware, and we train policies for this controller using a mu lti-criteria reward signal depending on both application performance and CPU power consumption. Our testbed scenario posed a number of challenges to successful use of RL, in- cluding multiple disparate reward functions, limited deci sion sampling rates, and pathologies arising when using multiple sensor readings as state variables. We describe innovative practical solutions to these challeng es, and demonstrate clear performance improvements over both hand-designed policies as well as obvious "cookbook" RL implementations.