Digital twin-based online resilience scheduling for microgrids: An approach combining imitative learning and deep reinforcement learning

Strong uncertainty of renewables puts high demands on the fast response of flexibility resources and resilience-oriented optimal scheduling for microgrids (MGs). Digital twins (DT) technology based on data-driven methods is a potential solution to this problem. A DT-based online resilience scheduling framework for MGs is designed in this study. Based on the proposed framework, a hybrid sequential-parallel combination method of imitation learning (IL) and deep reinforcement learning (DRL) is proposed to develop the optimal scheduling strategy for MGs. First, a mixed integer second-order cone programming (MISOCP) model is adopted to behave as an expert to generate decision demonstrations corresponding to operation scenarios of MGs. Then, IL and deep deterministic policy gradient (DDPG) are combined by (1) sequential pretrain and finetune and (2) parallel experience storing and sampling two steps to learn the optimal policy for MGs. Expert demonstrations from the MISOCP model are utilized to pretrain a deep neural network model by IL and initialize the policy network of DDPG to avoid it learning from scratch. Moreover, an expert replay buffer is introduced specifically to avoid forgetting the well-behaved expert experience and to further accelerate the training process. A 6-bus MG test system case study demonstrates the effectiveness and scalability of the proposed approach. A hybrid sequential-parallel combination method of imitation learning (IL) and deep reinforcement learning is proposed to learn the optimal policy. IL and deep deterministic policy gradient are combined to enhance the training performance by two steps: (1) sequential pretrain and finetune and (2) parallel experience storing and sampling. image