Assessing Health System Interventions: Key Points When Considering the Value of randomization/Evaluation Des Interventions Des Systemes De Sante: Points-Cles Lors De L'examen De la Valeur De la randomisation/La Evaluacion De Las Intervenciones En Sistemas Sanitarios: Aspectos Clave Al Considerar El Valor De la Aleatorizacion

Introduction Researchers are being urged to provide evidence on how to fix health systems in developing countries. (1-3) These exhortations recognize that health systems play vital role in achieving global goals for maternal, neonatal and child survival and for reducing HIV infection, tuberculosis and malaria. The type of research providing the best evidence on the effectiveness of health system interventions is matter of controversy, with quantitative and qualitative approaches often pitted against each other, although researchers are increasingly aware of the limitations of randomized studies (4,5) and of the value of mixed methods approaches. (6-8) Despite this, researchers who are better acquainted with individually randomized controlled trials (RCTs) than with other research designs still place undue reliance on randomization, particularly in health services research. Most health-care researchers understand that randomization eliminates or reduces bias and baseline imbalances between the groups being compared, and that the control group provides the comparison for the intervention under study. Clear reporting guidelines (9) have helped establish randomization as defining feature of a good intervention trial, concept that extends to cluster randomized designs. We agree that randomization is an extremely important tool in the researcher's armoury and do not dispute its importance in reducing the effects of various types of bias and confounding, especially when combined with concealment and blinding. These benefits are readily apparent when specific interventions, such as new drugs or vaccines, are tested at the individual level in safety, efficacy or effectiveness studies. Yet despite its undisputed value, randomization may not automatically provide the expected safeguards against confounding and bias, especially in research on what Lilford et al. have termed targeted or generic service interventions. (4) To help the general reader understand why the normal benefits of randomization are potentially reduced in the study of interventions delivered to components of the health system rather than directly to individuals we offer six points to consider. These points are also intended to illustrate the pitfalls of relying on the results of RCTs alone, without additional approaches to enquiry. Point 1: numbers As we try to examine larger units of health care delivery, fewer units are available for randomization. RCTs were designed to randomize large numbers of people into receiving either the intervention being tested or placebo. However, interventions targeting the health system are delivered not to individuals, but to groups, clinics, facilities or even larger units of organization such as districts. The larger the organizational unit, the fewer the units to be randomized, the larger the geographic area spanned by each unit and the greater the number of stakeholders involved, particularly if the study is of long duration. Feasibility then tends to constrain sample size. Unfortunately, if we the sample and intervene at given organizational level (a clinic, for example), we also need to randomize and to compare the results at that level (cluster). We can measure effects on clinic users, but these observations take place within cluster, and within cluster or clinic there are likely to be similarities in how people behave or are treated, thus the observations made within clinic are not entirely independent but may be influenced to greater or lesser degree by characteristics of the clinic (a point often overlooked). (10) Consequently, it may not be helpful to perform large number of observations within clinic (or cluster), as additional within-cluster recruitment typically yields diminishing returns. (11) Because the cluster is the unit of analysis, limited ability to recruit units reduces study power considerably, an effect for which it is seldom possible to compensate by increasing the number of within-unit observations. …