Analysis of Learning Outcomes in Software Engineering: an Automated Reflection Analysis Tool.

The rapid advancements in artificial intelligence (AI) have been transforming various domains, including engineering education. The availability of AI-based content and easy access to information have made students more dependent on these technologies. With the rise of online courses, there is growing concern about student engagement, poor learning outcomes, and low retention rates in higher education. Engagement plays a critical role in student success and can be achieved through formative assessment, critical thinking, and reflective thinking strategies. Reflection plays a key role in developing critical thinking and meta-cognitive skills. According to the constructive alignment framework which is an outcome-driven approach, the teaching and assessment methods should be shaped around fulfilling the course learning outcomes. Although the idea of constructive alignment is not a new topic, the higher education sector has recently emphasized it at a large scale due to the diversity of new required skills for students to enter the 5th industrial revolution. In earlier work, we proposed an AI-based reflection analysis model that combines both aspects of learning outcome-based assessment and student engagement by applying ‘Minute Paper'. Minute paper is a formative assessment tool that helps the instructors identify the muddy points of the lesson and students' learning gaps as well as their learning outcomes by asking two questions at the end of each lecture (i.e., what they learned and what they didn't). In this work, we propose a more advanced version of the reflection analysis tool by applying transformer-based language models to analyze students' responses to the Minute Paper reflections with higher accuracy in the course context. For this purpose, we train the BERTopic model with the course syllabus and lecture material to get more accurate data in the context of the given courses. The proposed system aids instructors in future course development by adding additional resources for the muddy points, allowing for adjusting content delivery pace, and designing tests and assignments with appropriate challenge levels [1]. Adaptation of this system can enhance the learning experience for both students and teachers and can be extended beyond higher education.