On the resilience of representative and novel data formats in CNNs

In recent years, a wide range of data type representations have been employed for training and storing the parameters of Deep Neural Networks (DNNs). The decision to employ a particular data type over another is influenced by various requirements, including the desire to enhance training accuracy or reduce data size to minimize memory usage, energy and power consumption. However, opting for one data type over another inevitably impacts the reliability of the model. This work studies the impact of different data representations on the reliability of LeNet-5, a popular Convolutional Neural Network (CNN) used for image classification tasks. An investigation is performed to evaluate the efficacy of the Average Bit-Flip Distance (ABFD) in predicting the criticality of bit positions in the data representation. The data type under analysis are FP32, POSIT32, POSIT16 and INT8. Together with the widely adopted metrics, this work proposes a new metric, called Soft SDC-n, to measure the percentage of faults that cause a change in the order of the top-n output elements. Experimental results shows that POSIT is not as reliable as FP32, while indicating that the most reliable data type is INT8. Furthermore, the same results confirm the presence of a relationship between the ABFD and the criticality of a bit in all the data representations under analysis.