ChatGPT's capabilities in providing feedback on undergraduate students' argumentation: A case study

In argumentation teaching, providing timely and high-quality feedback is always a challenging task for teachers because of the high complexity and large volume of students' argumentation contents. ChatGPT, a large language model introduced in November 2022, offers a potential solution for this problem. To examine the potential reliability and credibility of leveraging ChatGPT for argumentation feedback, the study conducted a retrospective analysis by applying ChatGPT to generate feedback on 50 sets of argumentation contents that human teachers had previously assessed. The study first assessed the feedback accuracy of ChatGPT and the factors that influenced the evaluation of arguments. The findings showed that ChatGPT demonstrated impressive precision rate (91.8 %) and recall rate (63.2 %) when providing feedback on arguments, indicating that ChatGPT possesses a fundamental capability to provide feedback on arguments. However, this capability of ChatGPT was significantly affected by the length of arguments and the discourse markers used in the arguments. The study then qualitatively compared the ChatGPT's feedback and teacher's feedback. The results revealed that these two types of feedback each had their own advantages and disadvantages. While ChatGPT could potentially generate comprehensive feedback and textual-based feedback, and limited to the linguistic level when provide affective feedback, teacher's feedback was more focused on student's overall learning progress, based on personal teaching experience to correctly identify immediate critical problem of the student, and consideration on the humanistic empathy interaction. Although the overall findings suggested that ChatGPT exhibited potential reliability and credibility for argumentation feedback, the study did identify several limitations.