Asynchronous deep reinforcement learning with gradient sharing for state of charge balancing of multiple batteries in cyber-physical electric vehicles

This work targets the State of Charge (SoC) imbalance issue due to the mismatch across multiple batteries in cyber–physical electric vehicles (EVs) arisen from a variety of practical factors such as manufacturing tolerance, nonuniform aging process and uneven operation temperature. While most of existing SoC balancing approaches are model-based and require accurate prior knowledge, a data-driven asynchronous deep reinforcement learning (DRL) with gradient sharing is proposed to equilibrate the SoC of multiple battery packs in cyber–physical EVs under time-varying operating conditions, and the transmission of control signals in the EVs is formulated by ISO/IEC 15118 standards to ensure the security of the cyber–physical EVs. The gradient sharing (GS) based exploration scheme can expand the exploration space to diversify the actor policies in the training process and in turn reduce the entropy loss value and guarantee the convergence to the optimal policy in the long term. The proposed asynchronous advantage actor-critic with gradient sharing (A3C-GS) based SoC balancing approach trains the agents based on the output data of the multiple bidirectional DC-DC converter buck-boost circuit to avoid the use of vehicle dynamics and cumbersome operating modes that are difficult to accurately model. Furthermore, the DC bus voltage regulation is integrated simultaneously with the SoC balancing scheme. Comparison results based on simulating three/six batteries based EV system show that the proposed A3C-GS based SoC balancing scheme can achieve the highest steady-state rewards while the oscillation in the early training stage is larger resulted from a larger exploration range.