PhrasIS: Phrase Inference and Similarity Benchmark

We present PhrasIS, a dataset of Phrase pairs with Inference and Similarity annotations for the evaluation of semantic representations. This dataset fills the gap between word and sentence-level datasets, allowing to evaluate compositional models at a finer granularity than sentences. Contrary to other datasets, the phrase pairs are extracted from naturally occurring text in image captions and news, and were annotated by experts. We analyze the dataset, showing the relation between inference labels and similarity scores, and evaluated several well-known techniques obtaining satisfactory performance. The gap with respect to annotator agreement shows that there is plenty of room for improvement. In addition, we introduce the use of similarity and relatedness inference relations, showing that they are useful for inference. With 10K phrase pairs split in development and test, the dataset is an excellent benchmark for testing meaning representation systems.