Inclusivity Bugs in Online Courseware: A Field Study

Motivation: Although asynchronous online CS courses have enabled more diverse populations to access CS higher education, research shows that online CS-ed is far from inclusive, with women and other underrepresented groups continuing to face inclusion gaps. Worse, diversity/inclusion research in CS-ed has largely overlooked the online courseware-the web pages and course materials that populate the online learning platforms-that constitute asynchronous online CS-ed's only mechanism of course delivery. Objective: To investigate this aspect of CS-ed's inclusivity, we conducted a three-phase field study with online CS faculty, with three research questions: (1) whether, how, and where online CSed's courseware has inclusivity bugs; (2) whether an automated tool can detect them; and (3) how online CS faculty would make use of such a tool. Method: In the study's first phase, we facilitated online CS faculty members' use of GenderMag (an inclusive design method) on two online CS courses to find their own courseware's inclusivity bugs. In the second phase, we used a variant of the GenderMag Automated Inclusivity Detector (AID) tool to automatically locate a "vertical slice" of such courseware inclusivity bugs, and evaluated the tool's accuracy. In the third phase, we investigated how online CS faculty used the tool to find inclusivity bugs in their own courseware. Results: The results revealed 29 inclusivity bugs spanning 6 categories in the online courseware of 9 online CS courses; showed that the tool achieved an accuracy of 75% at finding such bugs; and revealed new insights into how a tool could help online CS faculty uncover assumptions about their own courseware to make it more inclusive. Implications: As the first study to investigate the presence and types of cognitive- and gender-inclusivity bugs in online CS courseware and whether an automated tool can find them, our results reveal new possibilities for how to make online CS education a more inclusive virtual environment for gender-diverse students.