Designing an instrument to measure the development of techno-mathematical literacies in an innovative mathematics course for future engineers in STEM education

Techno-mathematical Literacies (TmL), which are defined as a combination of mathematical, workplace and ICT knowledge, and communicative skills, are acknowledged as important learning goals in STEM education. Still, much remains unknown about ways to address them in teaching and to assess their development. To investigate this, we designed and implemented an innovative course in applied mathematics with a focus on Techno-mathematical Literacies for 1st-year engineering students, and we set out to measure the learning effect of the course. Because measuring TmL is an uncharted terrain, we designed tests that could serve as pre- or posttests. To prevent a test learning effect, we aimed to design two different but equally difficult tests A and B. These were assigned randomly to 68 chemistry students, as a pretest, with the other one serving as posttest after the course. A significant development in TmL was found in the B-pre group, but not in the A-pre group. Therefore, as a follow-up analysis we investigated whether the two tests were equally difficult and searched for possible explanations. We found that test B was indeed perceived as more difficult than test A, but also that students who were assigned B (pre) were previously higher achieving than A (pre), and a sound mastery level of basic skills that ground the higher-order TmL seemed necessary. Furthermore, as TmL are very heterogenous by nature, some of them are easier learned and measured than others. Based on the results, we propose ways of testing TmL, which should be validated in future research.