Model-Based Systems Thinking: Assessing Engineering Student Teams

Contribution: A rubric for assessing the systems thinking expressed in conceptual models of technological systems has been constructed and assessed using a formal methodology. The rubric, a synthesis of prior findings in science and engineering education, forms a framework for improving communication between science and engineering educators. Background: Systems thinking is an important skill in engineering, but to date no rubric for assessing this skill based on a formal methodology has been published with reliability and validity measures. Research Questions: (a) What attributes should a rubric for assessing the systems thinking of engineering students comprise? (b) To what extent can such a rubric serve in assessing the systems thinking level of engineering students, as expressed in their conceptual system models? Methodology: Based on a literature review of systems thinking assessment in science and engineering education, the authors classified the systems thinking attributes they had identified into system function, structure, and behavior. Scoring instructions were developed for conceptual system models based on Object-Process Methodology ISO 19450, a formal methodology and language for model-based systems engineering. A total of 142 undergraduate engineering students in 32 teams modeled Web-based systems using the methodology. Each team submitted two models of the same system, at the middle and end of semester. Models were scored using the rubric, and its reliability and validity were evaluated. Findings: Indications of interrater reliability, internal consistency, and construct validity were acceptable, excluding the system function aspect, implying the rubric may be used reliably for its intended purpose.