Dynamic Resistance Based Spiking Actor Network for Improving Reinforcement Learning.

Designing algorithms for continuous control is a big challenge no matter in real robot controlling or simulation tasks. Deep reinforcement learning (DRL) is the most successful algorithm in dealing with such tasks for it utilizing the powerful ability of deep neuron network (DNN) in handling complex information. However, the more powerful ability the DNN has, the more energy it consumes. It’s a barrier for DRL to be realized in real-world control tasks. With more biological features, spiking neuron network (SNN) is one of the frontier fields of high-efficiency computing. The binary spike it used to represent information contains more temporal information and leads to greater computational efficiency on neuromorphic chips. Based on a hybrid architecture of SNN and DNN, we propose an actor-critic model to utilize the ability of SNN in dealing with complex continuous information and the ability of DNN in large scale accurate computation. The common Leaky Integrate-and-Fire (LIF) neuron model which is mainly used to build deep SNN neglects the resistance flexibility in the neuron. Considering that causes a descend capacity of representing continuous information which is of vital important in continuous control, we propose a new dynamic resistance LIF (R-LIF) model to compensate the temporal relation dependencies in neurons. With the same gradient updating rule, our R-LIF based spiking actor network (RSAN) shows a better performance when inferring in OpenAI benchmark tasks not only than the deep neuron actor network but also than the same LIF based spiking neuron actor network.