Discovering and Quantifying Misconceptions in Formal Methods Using Intelligent Tutoring Systems

In this paper we advocate the study of misconceptions in the formal methods domain by integrating quantitative and qualitative methods. In this domain, so far, misconceptions have mostly been studied with qualitative methods, typically via interviews with less than 20 subjects. We discuss workflows for (1) determining the commonness of qualitatively established misconceptions by quantitative means; and for (2) the initial discovery of misconceptions by quantitative methods followed by qualitative assessments. Parts of these workflows are then applied to a data set for exercises on logical modeling from the intelligent tutoring system ILTIS with > 250 data points for many of the exercises. We analyze the data in order to (1) determine the commonness of qualitatively-identified misconceptions on modeling in propositional logic; and to (2) discover typical mistakes in modeling in propositional logic, modal logic, and first-order logic.