Automated Feedback, the Next Generation: Designing Learning Experiences

ABSTRACTThe SIGCSE Technical Symposium is a wonderful venue for learning about innovative tools of all sorts for teaching and learning programming. We are at the cusp of a new focus for tool research and development, and this presentation proposes a vision for the next generation of teaching tools, using automated program assessment as an example for how our tools should evolve. Beginning in the 1960s, the first generation of automated program grading tools focused on workflow automation. Automating the processing of student work to increase the speed and accuracy of mechanical steps would reduce the time human graders spend on these tasks, allowing them to be more effective at the more meaningful aspects of grading. After mastering automation, the second generation of autograders focused on assessment--specifically, increasingly better strategies for evaluating the quality and correctness of student work. The last two decades have seen significant work in this area, resulting in a change in terms to "automated program assessment tools." Now, we are beginning to see a new focus emerge, which will lead to a new generation of tools for teaching and learning. While assessing the final product that a student creates is the essence of the "grading" task, our goal as educators is not to grade, but to teach. One great benefit of automated program assessment tools is that they provide feedback to students with the aim of helping students to improve their work and to learn. Indeed, the assessment tool and the student form a closed-loop feedback system, and we should not ignore the role the human plays in that system. It is not enough to objectively and accurately assess the work product--we must consider how the feedback itself and the way it is delivered affect the student's experience, with the goal of contributing to student learning. The student's experience in receiving feedback can affect their beliefs, their motivation, and their behavior. Instead of focusing primarily on the work product (the solution being produced), tool researchers need to broaden their perspectives to adopt specific goals for student learning that their tools will promote. Improving self-efficacy, encouraging a growth mindset, encouraging students who are challenged, recognizing accomplishments, and fostering active engagement are all aspects a human grader may consider when providing feedback by hand, and the same concerns should inform the design of automated feedback. A holistic view of how the feedback affects the learner is essential. As a result, I advocate moving from a tool-centric perspective toward designing complete learning experiences, where the student interacting with a teaching tool through feedback is explicitly considered. The primary goal is to foster targeted changes in specific beliefs, behaviors, skills, and knowledge in the student, rather than just to "grade" their work. By focusing on the student experience and the intended changes (or learning or growth) in the student as the goal, we can reframe the nature of "assessment" tools in a way that will change our outlook on what can be accomplished.