Information Theoretic Probing with Minimum Description Length
EMNLP 2020, pp. 183-196, 2020.
We propose information-theoretic probing which measures minimum description length of labels given representations
To measure how well pretrained representations encode some linguistic property, it is common to use accuracy of a probe, i.e. a classifier trained to predict the property from the representations. Despite widespread adoption of probes, differences in their accuracy fail to adequately reflect differences in representations. For example, th...More
PPT (Upload PPT)