conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks
CoRR(2024)
摘要
Modern ConvNets continue to achieve state-of-the-art results over a vast
array of vision and image classification tasks, but at the cost of increasing
parameters. One strategy for compactifying a network without sacrificing much
expressive power is to reshape it into a tensorial neural network (TNN), which
is a higher-order tensorization of its layers, followed by a factorization,
such as a CP-decomposition, which strips a weight down to its critical basis
components. Passes through TNNs can be represented as sequences of multilinear
operations (MLOs), where the evaluation path can greatly affect the number of
floating point operations (FLOPs) incurred. While functions such as the popular
einsum can evaluate simple MLOs such as contractions, existing implementations
cannot process multi-way convolutions, resulting in scant assessments of how
optimal evaluation paths through tensorized convolutional layers can improve
training speed. In this paper, we develop a unifying framework for representing
tensorial convolution layers as einsum-like strings and a meta-algorithm
conv_einsum which is able to evaluate these strings in a FLOPs-minimizing
manner. Comprehensive experiments, using our open-source implementation, over a
wide range of models, tensor decompositions, and diverse tasks, demonstrate
that conv_einsum significantly increases both computational and
memory-efficiency of convolutional TNNs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要