On the Discriminability of Samples Using Binarized ReLU Activations

Binarized ReLU activations are considered as a metric space equipped with the Hamming distance. While for two-layer ReLU networks with random Gaussian weights it can be shown theoretically that local metric properties are approximately preserved, we experimentally study the discrimination capability in this Hamming space for deeper ReLU networks and look also at the non-local behavior. It turns out that the discrimination capability is approximately preserved as expected.