Evaluating Average and Heterogeneous Treatment Effects In Light of Domain Knowledge: Impact of Behaviors on Disease Prevalence

Understanding causal treatment effect and its heterogeneity can improve targeting of efforts for prevention and treatment of diseases. A number of methods are emerging to estimate heterogeneous treatment effect from observational data, such as Causal Forest. In this paper, we evaluate the heterogeneous treatment effect estimates in terms of whether they recover the expected direction of the effect based on domain knowledge. We use the individual level health surveys conducted by the Turkish Statistical Institute (TUIK) over the span of eight years with 90K+ observations. We estimate the effect of six behaviors on the probability of two diseases (IHD and Diabetes). We compare two approaches: a) treatment and disease specific Causal Forest models that directly estimate the heterogeneous treatment effect, and b) disease specific Random Forest models of disease probability that are used as simulators to evaluate counterfactual scenarios. We find that, with some exceptions, the signs of Causal Forest heterogeneous treatment effects are aligned with domain knowledge. Causal Forest performed better than the more naïve approach of using RF models as simulators which disregards selection bias in treatment assignment.