Beyond First Encounters with Analytics - Questions, Techniques and Challenges in Instructors' Sensemaking.

Despite growing implementation of teacher-facing analytics in higher education, relatively little is known about the detailed processes through which instructors make sense of analytics in their teaching practices beyond their initial encounters with tools. This study unpacked the sensemaking process of thirteen instructors with analytic experience, using interviews that included walk-throughs of their analytics use. Qualitative inductive analysis was used to identify themes related to (1) the questions they asked of the analytics, (2) the techniques they used to interpret them, and (3) the challenges they encountered. Findings indicated that instructors went beyond a general curiosity to develop three types of questions of the analytics (goal-oriented, problem-oriented, and instruction modification questions). Instructors also used specific techniques to read and explain data by (a) developing expectations about the answers the analytics would provide, and (b) making comparisons to reveal student diversity, identify effects of instructional revision and diagnose issues. The study found instructors faced an initial learning curve when seeking and making use of relevant information, but also continued to revisit these challenges when they were not able to develop a routine of analytics use. These findings both contribute to a conceptual understanding of instructor analytic sensemaking and have practical implications for its systematic support.