A Systematic Literature Review of Methodology of Learning Evaluation Based on Item Response Theory in the Context of Programming Teaching

This Research Full Paper presents a Systematic Literature Review (SLR) that investigates state-of-the-art methods, processes, approaches, and instruments based on Item Response Theory (IRT) in the programming teaching context. Various studies support professors and students to improve the evaluation process in teaching programming. Among the different techniques and theories associated with these studies, the IRT gained prominence because of its effectiveness. Due to the lack of an overview in the area, we present a SLR with the objective of identity studies that use IRT as a methodology of evaluation of learning in the context of Programming Teaching; studies which present real-time feedback; and experimental studies that have been carried out to validate them. To achieve these goals, we planned an SLR following the guidelines proposed by Kitchenham (2004). Our main findings are a) There is a limited number of studies that explore IRT in evaluation methodologies in the teaching programming context. Among these studies, the main focus was the development on of instruments to measure programming skills; b) Most instruments are multiple-choice, adopt the 3PL model, and do not show evidence of real-time feedback except SIETTE and CodeWorkout and are also the only instruments that evaluate the coding capacity of individuals; c) The studies showed scientific evidence on the use of IRT in the evaluation methodology in the programming teaching context. The studies presented experiments that indicate a significant gain in the measurement of programming abilities when compared to the traditional evaluation method.