Evaluating the Code Comprehension of Novices with Eye Tracking

Code comprehension is essential for software evolution, but it can be hindered by structural problems in the code. Despite its importance, researchers often rely on static code metrics to investigate code comprehension, without considering dynamic aspects such as the visual effort involved. Therefore, we aim to fill this gap by examining code comprehension using eye-tracking metrics. We conducted three controlled eye-tracking studies with a total of 128 novices in three distinct programming languages: C, Python, and Java. We used constructions with different granularity levels, namely atoms of confusion, refactorings, and #ifdef annotations. With respect to atoms of confusion, we found that the clarified version of the code reduced the time and number of attempts by 38.6% and 28%, respectively. The use of the Extract Method refactoring significantly reduced the time of two tasks by 70% to 78.8%, while increasing the accuracy of three tasks by 20% to 34.4%. The use of #ifdef annotations presented reductions in the modified region by 46.9% in time, 44.7% in the fixation duration, 48.4% in the fixation count, and 60.5% in the regressions count. In addition, we also identified patterns in our data by observing several visual transitions going forward and backward between specific code elements which allowed us to infer confusion in the code, such as when adding the particle not before the variable. These results provide educators, practitioners, and researchers with valuable information on certain transformations and their potential to ease or hinder code comprehension. The study also highlights the potential of visual metrics to reveal the impact of transformations that cannot be captured by static code metrics. Our results also pave the way for the development of more advanced tools that track the eye movements of developers and assist them with helpful suggestions.