An evaluation of the quality of interviewer and virtual observations and their value for potential nonresponse bias reduction

With the decline of survey response rates over the past decade, survey researchers need to gather useful auxiliary variables for all sampled units and reduce the potential for nonresponse bias through adaptive survey design or nonresponse weighting adjustments. One potential source of auxiliary information is interviewer observations of characteristics of sampled units. Compared with area-level characteristics, which researchers often have available, characteristics at the dwelling unit level may provide more information about survey variables of interest and result in weight adjustments that could potentially reduce bias further. These observations, however, may vary greatly among observers, and may lack the quality needed for survey data producers. To investigate the quality and usefulness of such observations, this study systematically assesses completeness, validity, observer variance/reliability, and predictive power for bias reduction in a national pilot study for both in-person interviewer observations and virtual observations. This paper sheds light on the dwelling unit characteristics that are harder to observe, differences among interviewer and virtual observations, the potential value added beyond area-level characteristics for adaptive survey design and nonresponse adjustments, and ways to improve the observations.