We'll fix it in post: what do bug fixes in video game update notes tell us?

ABSTRACTBugs that persist into releases of video games can have negative impact on both developers and users. Despite these impacts, it is common for games to release with bugs that are fixed through subsequent updates. In 2014, for example, over a third of big-budget games "released on Xbox One, Wii U and PS4" received an update within 24 hours of the game's initial release [5]. The differences between the development of games and development of traditional software could explain the appearance of these bugs. Prior research has indicated that there is considerable difficulty in comprehensively testing all aspects of a video game [4], [7]. Developers have difficulty writing comprehensive tests, because games can have a significantly large number of possible user interactions compared to other types of software [7]. As a result, many games release with undiscovered bugs that only reveal themselves once customers begin playing the game [5]. It has become common practice for developers to apply updates to games in order to fix missed bugs [3]. These updates are often accompanied by notes that describe the changes to the game included in the update [8]. However, some bugs might recur across multiple updates. There exist cases in which developers have attempted to fix a bug in one update, only for that bug to reappear in the game anyway, necessitating another attempted fix in a later update [1]. Previous research has focused on creating a taxonomy for video game bugs [2]. To address shortcomings of this prior taxonomy and to attain a deeper understanding of the types of bugs in games, we expand the taxonomy of bug types. We analyzed 12,122 bug fixes taken from 723 updates for 30 popular games on the Steam platform. We categorized these bug fixes using our taxonomy of bug types. We then analyzed the frequency at which the different bug types appear in the update notes and investigated which types of bugs recur more often over multiple updates. Additionally, we investigated which types of bugs most frequently appear in urgent updates or hotfixes, as the bugs that appear in these updates are more likely to have a severe negative impact on users [8]. Finally, we surveyed game developers on their experience with these different types of bugs as well as what challenges and techniques are involved in fixing these bugs. For the recurrence analysis, we used an automated approach to identify potential recurring bug fixes, then manually evaluated these fixes to find the true recurring bugs. We first performed cosine similarity analysis [6] between bug fix lines. For each bug fix line from a game's updates, we compared that line to each line in all subsequent updates for the game. If two lines had a similarity score of at least 90%, they were treated as a potential match and grouped together for manual review. To determine bug severity, we applied a combination of metrics to identify urgent updates, including some metrics employed by past researchers [3]. Once we identified all the urgent updates in our data, we flagged all the bug fixes lines from our data that appeared in an urgent update. An urgent update is generally intended to fix "problems that are deemed critical enough to not be left unfixed until a regular-cycle update" [3]. We calculated the severity of each bug type by finding the proportion of bug fixes from the data that appeared in an update marked as urgent. For each bug type, we divided the frequency of the bug type in an urgent update by the frequency of the bug type in all updates. Fig. 1. Frequency of Bug Types Across All Updates The most frequently occurring bug types were Information, Game Graphics, and Action (Figure 1). The bug types that recurred the most frequently over multiple updates were Crash, Game Graphics, and Triggered Event (Figure 2). Based on the update data, the bug type with the highest severity was Crash. The next most severe bug types were Object Persistence and Triggered Event (Figure 3). Meanwhile, from the survey, the bug types that were deemed to have the greatest severity were Crash, Action, and Exploit (Figure 4). Fig. 2. Frequency of Recurring Bug Types Fig. 3. Severity of Bug Types We received 47 applicable responses to our survey. According to respondents, the aspects of game development most frequently linked to bug recurrence were testing, game design, and code quality. With testing, responses ranged from dealing with a lack of testing in general to dealing with a lack of certain kinds of testing, such as automated testing, integration testing, and cross-platform testing. Additionally, the most frequently mentioned main challenges to identifying and fixing bugs in video games were inadequate testing, reproducing bugs, and code quality. With respect to code quality, there was a particular emphasis on the importance of well-written code that does not cause conflicts elsewhere in the game. Fig. 4. Agreement Level to Whether Selected Bug Types are Likely to Have a Severe Negative Impact on Game Experience These results can help game developers identify which types of bugs to pay more attention to when testing and fixing bugs. Developers can also use these results to help adjust practices related to game development process in order to better prevent, identify, and fix bugs. Researchers can take advantage of these results in order to develop tools or methods that can target specific bug types that are more likely to severely impact the game. There is room for future work that can identify aspects of game development that might benefit from specialized tools or methods that address some of the challenges provided in the survey. For example, multi-component interaction testing is one area that could benefit from future research. Our study takes the first step towards fulfilling these goals.