Inner Product Accelerating Scheme Based on RRAM Array for Attention-Mechanism Neural Network

The inner product between two activation vectors is crucial for implementing neural networks with the attention mechanism. In this work, we propose and experimentally validate a novel inner product accelerating scheme based on the resistive random access memory (RRAM) array. By utilizing the RRAM conductance to represent the significance of each bit, the inner product can be conducted with a high energy efficiency of analog computing. Furthermore, an attention-mechanism neural network is demonstrated for solving the Q&A task in the Facebook bAbI dataset. We verify software-comparable accuracy (98.5%) while achieving a 42-fold reduction in latency and a 71-fold reduction in energy consumption compared with the conventional digital platform.