When Test Cases Are Not Enough: Identification, Assessment, and Rationale of Misconceptions in Correct Code (MC³)

Automated grading systems (autograders) assist the process of teaching in introductory programming courses (CS1). However, the sole focus on correctness can obfuscate the assessment of other characteristics present in code. In this work, we investigated if code, deemed correct by an autograder, were developed with characteristics that indicated potential misunderstandings of the concepts taught in CS1. These characteristics were nominated Misconceptions in Correct Code (MC³). By analyzing 2,441 codes developed by CS1 students, we curated an initial list of 45 MC³. This list was assessed by CS1 instructors, resulting in the identification of the misconceptions that most needed addressing in classes. We selected the 15 most severe MC³ for further investigation, including a semi-structured observation in a CS1 course and a prototype of an automated detection using static analysis of code. The obtained results suggested that students develop these misconceptions either due to an incomplete comprehension of the concepts taught in the CS1 course or a lack of attention while elaborating their code, with correctness being their primary goal. We believe our results can contribute to: (1) the research field of misconceptions in CS1; (2) promoting alternative approaches to complement the use of autograders in CS1 classes; and (3) providing insights that can serve as the foundation for teaching interventions involving MC³ in CS1.