Do You Know That Florence Is Packed With Visitors? Evaluating State-Of-The-Art Models Of Speaker Commitment

When a speaker, Mary, asks Do you know that Florence is packed with visitors?, we take her to believe that Florence is packed with visitors, but not if she asks Do you think that Florence is packed with visitors? Inferring speaker commitment (aka event factuality) is crucial for information extraction and question answering. Here we explore the hypothesis that linguistic deficits drive the error patterns of speaker commitment models by analyzing the linguistic correlates of model errors on a challenging naturalistic dataset. We evaluate two state-of-the-art speaker commitment models on the CommitmentBank, an English dataset of naturally occurring discourses. The CommitmentBank is annotated with speaker commitment towards the content of the complement (Florence is packed with visitors in our example) of clause-embedding verbs (know, think) under four entailment-canceling environments. We found that a linguistically-informed model outperforms a LSTM-based one, suggesting that linguistic knowledge is needed to capture such challenging naturalistic data. A breakdown of items by linguistic features reveals asymmetrical error patterns: while the models achieve good performance on some classes (e.g., negation), they fail to generalize to the diverse linguistic constructions (e.g., conditionals) in natural language, highlighting directions for improvement.