Multi-agent Hierarchical Deep Reinforcement Learning for Operation Optimization of Grid-interactive Efficient Commercial Buildings

The operation optimization of grid-interactive efficient commercial buildings (GECBs) contributes to overcoming the issues caused by increasing peak electricity demand. However, existing GECB operation optimization methods either rely on prior information of stochastic parameters or do not consider the multi-timescale characteristics of heating, ventilation, and air conditioning (HVAC) systems. In this paper, we investigate a GECB operation optimization problem considering HVAC multi-timescale characteristics. Specifically, an expected GECB operation cost minimization problem is first formulated. To overcome the challenges in solving the formulated problem (e.g., uncertain parameters, inexplicit system models, multiple timescales, and high-dimensional discrete solution space), we reformulate the above problem as a Markov game and two Markov decision processes. Then, we design a GECB operation optimization algorithm based on multi-agent hierarchical deep reinforcement learning, which can support two kinds of smart coordination, i.e., the coordination among agents at slow timescales, and the coordination among agents at different timescales. Performance evaluation with real-world data-sets indicates that the designed algorithm could decrease total operation cost by 6.16%-45.52% while maintaining comfortable indoor environments and efficient grid services compared with adopted benchmarks.