Evaluating the development of chemistry undergraduate researchers’ scientific thinking skills using performance-data: first findings from the performance assessment of undergraduate research (PURE) instrument

National calls in science, technology, engineering, and technology education reform efforts have advanced the wide-scale engagement of students in undergraduate research for the preparation of a workforce and citizenry able to attend to the challenges of the 21st century. Awareness of the potential benefits and costs of these experiences has led to an emerging literature base outlining gains in participants' cognitive, affective, and conative domains to support the impact of undergraduate research for students of all backgrounds; however, the majority of this work has relied on self-report data limiting inferences to the causal effects on student learning. As part of a larger project on apprentice-like undergraduate research experiences (UREs) in the physical sciences, the present exploratory study complemented indirect self-report data with direct performance data to assess the development of chemistry students' scientific thinking skills over a research experience. Performance data were collected using the Performance assessment of Undergraduate Research Experiences (PURE) instrument, a validated tool designed to assess changes in chemistry students' analytical and data driven decision-making skills through open-response tasks situated in real-world problems from primary literature. Twenty-four summer research students in chemistry (46% women; 50% 1st/2nd year students; 42% first time URE participant) from seven colleges and universities provided baseline and post-intervention performance data. Differences in pre/post-response task correctness provided a direct measure of individual changes in student competencies. Early study findings indicate the positive contributions of UREs to student's competencies in the areas of problem-solving, experimental design and the use of research techniques, data analysis and the interpretation of results, and the evaluation of primary literature. Survey data were also collected on students' self-skill ratings to allow comparisons between perceived and demonstrated competencies, which were found to be weakly correlated. This work begins to offer direct evidence to the effect of UREs on student learning progressions as well as the potential use of performance test data in evaluating the success of research training interventions designed to improve scientific thinking skills.