A comparison of Softmax proposals

A number of alternative proposals has recently appeared in the literature for implementing or approximating the Softmax function. We briefly review the basic ideas that are used in each proposal for optimizing the calculation and we attempt to provide a fair comparison of them by using the same set of assumptions and parameters, such as input and output size, value representations, etc. To this end, we present architectures for the implementation of each method, which are synthesized for attaining performance estimates and we study the introduced error.