Exploring Barriers in Productive Failure

Motivation and Objectives. Productive Failure is a problem-based learning technique where students attempt to solve a problem before receiving instruction in the topic. By design, students may not find a satisfying solution. Prior studies of Productive Failure in STEM contexts have been conducted in secondary or introductory college settings. Focusing primarily on exploring appropriate analysis and modeling techniques, these studies showed that a Productive Failure approach can lead to greater conceptual knowledge acquisition and transfer capabilities compared to "traditional"Direct Instruction techniques. In this study, we build on these studies along two dimensions: First, we report on the design and evaluation of a Productive Failure intervention in a more advanced undergraduate class: third-year Operating Systems. Second, our intervention targeted a more advanced skill: applying synchronization primitives, rather than selecting appropriate modeling and analysis techniques. Methods. We ran a quasi-experimental study in an undergraduate Operating Systems course to compare the effects of Productive Failure (PF) with Direct Instruction (DI) on students' learning. To ensure fidelity of implementation as well as to explore different modes of instruction, we ran a pilot study in a remote learning environment. The final study was conducted in an in-person classroom environment. We collected and analyzed both qualitative and quantitative data to gather insights into the students' problem-solving process and learning outcomes. Results. In line with previous studies, our study did not provide empirical evidence that there was any statistically significant difference with respect to reflection or short-term transfer. While we were able to verify that students in the Productive Failure condition explored a wider spectrum of solution approaches, we could not reliably detect different communication patterns across the conditions. Finally and unlike previous studies, our study was unable to detect longer-term difference in transfer capabilities. Discussion. We failed to observe several of the advantages of Productive Failure over Direct Instruction seen in prior work in other contexts. However, we note several differences in context relative to those earlier studies, including both the complexity of the topics covered and the type of intended learning outcomes. Consequently, our results do not directly contradict earlier findings. Still, they leave room for interpretation and call for further investigation of this instructional approach in more advanced computer science classes.