A Critical Evaluation of Tracking Surveys with Social Media: A Case Study in Presidential Approval

Relationships found between public opinion polls and data extracted from social media have led to optimism about supplementing traditional surveys with these new sources of data. However, many initial findings have not been met with usual levels of scrutiny and skepticism. Our goal is to introduce a higher level of scrutiny to these types of analyses. We focus on presidential approval, because we believe signals relating to politics might be some of the strongest in social media, and provide an illuminating test case. Our first contribution is to develop a framework to interpret the strength of relationships found between public opinion poll surveys and tweets containing a given keyword. Following methods that exist in the literature, we measure the association between survey based measures of presidential approval and tweets containing the word “Trump.” We then implement placebo analyses, in which we perform the same analysis as with the “Trump” tweets but with tweets unrelated to presidential approval, and we conclude that the relationship between “Trump” tweets and public opinion polls is not strong. As our second contribution, we suggest following social media users longitudinally. For a set of politically active Twitter users, we classify users as a Democrat or Republican and find evidence of a political signal in terms of frequency and sentiment of their tweets around the 2016 presidential election. However, even in this best-case scenario of focusing exclusively on politics and following users who are politically engaged, the signal found is relatively weak. For the goal of supplementing traditional surveys with data extracted from social media, these results are encouraging, but cautionary.