Post Recommendation System Impact on Student Participation and Performance in an Online AI Graduate Course.

Embedding a post recommendation system in online course forums improves students' ability to find relevant course content. Yet, there is limited research quantifying how these interventions impact forum interactions and students' class performance. We randomly divide an on-line class in a Masters degree-seeking program into control and experimental sections. Midway through the class, for the experimental section, we introduce an information retrieval system that suggests relevant posts to students while they compose their new posts. The average number of initial posts in the discussion forum dropped by 55% for the experimental group compared to 21% in the control group. In post-hoc analysis, we classify students as having an active or passive (i.e., mostly lurking) forum participation style. Posts per student per assignment by passive participants in the experimental group decreased by 15%, while passive participants' posts in the control group increased by 200%. The number of answers given by instructors in the experimental group decreased twice as much as the control group after intervention, though the difference was not statistically significant. The reduction of posts was not associated with a decrease in academic performance (grades) for the experimental group. This line of research convinced the popular forum Piazza to implement a post recommendation system similar to that used in these experiments.