Evaluation of BERT and XLNet Models on Irony Detection in English Tweets

Automatically detecting irony is helpful and important for mining fine-grained information from social web data. Therefore, the International Workshop on Semantic Evaluation (SemEval) presented the first shared task on irony detection called "Irony Detection in English Tweets" in 2018. For this task, the system should determine whether the tweet is ironic (Task A) and which type of irony is expressed (Task B). The top teams obtained 𝐹1=0.71 for Task A and 𝐹1=0.51 for Task B. These teams all used LSTM models exploiting some word embedding features like Glove embeddings. While in recent years more powerful models like BERT and XLNet appeared. Therefore, in our paper, we evaluate the performance of BERT and XLNet models on Irony Detection in English Tweets by two methods: word embedding method and fine-tuning method. Through the experiment, these two models could get relatively high scores showing that BERT and XLNet models are capable to understand the irony to some extent.