VisHikers’ Guide to Evaluation: Competing Considerations in Study Design

In this Viewpoint article, we describe the persistent tensions between various camps on the “right” way to conduct evaluations in visualization. Visualization as a field is the amalgamation of cognitive and perceptual sciences and computer graphics, among others. As a result, the relatively disjointed lineages in visualization understandably approach the topic of evaluation very differently. It is both a blessing and a curse to our field. It is a blessing, because the collaboration of diverse perspectives is the breeding ground of innovation. Yet it is a curse, because as a community, we have yet to resolve an appreciation for differing perspectives on the topic of evaluation. We explicate these differing expectations and conventions to appreciate the spectrum of evaluation design decisions. We describe some guiding questions that researchers may consider when designing evaluations to navigate differing readers’ evaluation expectations.