RF33 Evaluation of public health interventions using a complex systems lens: a critical review

Background There has been a growth in interest in applying systems thinking to public health research: including greater consideration of the complex and changing nature of real-world environments within which public health interventions take place. In this paper we present the results from a critical review that asked the question: how can a systems approach be applied in the context of public health evaluation? Methods A critical review of the literature was conducted to identify contrasting examples of systems approaches for in-depth comparison and analysis. To inform our protocol and identify relevant studies we held consultations with international researchers with relevant expertise (n=32). We tracked citations from previous reviews and searched Scopus, Medline and Web of Science from 01/01/14 to 06/08/17. We used search terms relating to systems and complexity, evaluation, public health and its social determinants. Our inclusion criteria were as follows: studies must (i) self-identify as taking a systems or complexity-informed approach; and (ii) evaluate one or more interventions or changes in a public health relevant field. Study selection, appraisal, data extraction and analysis were conducted independently by at least two reviewers with regular meetings to discuss contrasting viewpoints. Results Forty-four studies were finally included. Public health topics varied: the most common concerned obesity, transport, education, and tobacco. Evaluations were classified by the systems approach taken; in total 6 approaches were identified: qualitative research (n=13), concept mapping (n=3), network analysis (n=4), system dynamics modelling (n=15), agent-based modelling (n=8), and ‘systems friendly’ approaches (n=5). These different approaches were used to address different research questions but there was also cross-over between methodological approach and purpose. Some studies lacked clear empirical conclusions for informing future practice. Discussion Though sometimes portrayed as a novel development in public health research, there are already numerous examples of different public health ‘systems evaluations’. There is no single or dominant ‘systems approach’ to public health evaluation. Nor is there a consistent pattern whereby different approaches address research questions specific to that approach. Rather than advocate for a single approach to systems evaluation, we believe continued innovation in this field is most helpful at this time. To improve utility, some systems evaluations would benefit from improvements in the reporting of empirical findings.