Machine Learning Based Action Recognition with Modular CNN

When building models for action recognition, 3D convolutional neural networks (CNNs) are commonly used. However, 3D CNNs also increase the model parameters significantly. We propose two methods, image segmentation and dynamic sampling learning to reduce network parameters and required memory access. Using image segmentation to keep the location of the action and remove the background of each image reduces the size of the feature map. Dynamic sampling learning allows the model to learn from low sampling rates without adding additional parameters, and to maintain performance while reducing the number of images. In order to implement the overall model in hardware for edge devices, we limit the kernel sizes of the 2D convolution layers and 3D convolution layers in the model to only 3x3 and 3x3x3 respectively. We perform experiments on HMDB51 [1] and UCF101 [2] datasets respectively with our proposed model. The accuracy of our proposed method achieve 7.2% and 5.9% reduction compared with DS-GRU2021 [3]. However, the number of parameters of our model is 30% fewer and execution speed x180 faster than DS-GRU2021 [3].