'Shaking the Ladder' reveals how analytic choices can influence associations in nutrition epidemiology: beef intake and coronary heart disease as a case study

Background Many analytic decisions are made when analyzing an observational dataset, such as how to define an exposure or which covariates to include and how to configure them. Modelling the distribution of results for many analytic decisions may illuminate how instrumental decisions are on conclusions in nutrition epidemiology. Objective We explored how associations between self-reported dietary intake and a health outcome depend on different analytical decisions, using self-reported beef intake from a food frequency questionnaire and incident coronary heart disease as a case study. Design We used REasons for Geographic and Racial Differences in Stroke (REGARDS) data, and various selected covariates and their configurations from published literature to recapitulate common models used to assess associations between meat intake and health outcomes. We designed three model sets: in the first and second sets (self-reported beef intake modeled as continuous and quintile-defined, respectively), we randomly sampled 1,000,000 model specifications informed by choices used in the published literature, all sharing a consistent covariate base set. The third model set directly emulated existing covariate combinations. Results Few models (<1%) were statistically significant at p<0.05. More hazard ratio (HR) point estimates were >1 when beef was polychotomized via quintiles (95% of models) vs. continuous intake (79% of models). When covariates related to race or multivitamin use were included in models, HRs tended to be shifted towards the null with similar confidence interval widths compared to when they were not included. Models emulating existing published associations were all above HR of 1. Conclusions We quantitatively illustrated the impact that analytical decisions can have on HR distribution of nutrition-related exposure/outcome associations. For our case study, exposure configuration resulted in substantially different HR distributions, with inclusion or exclusion of some covariates being associated with higher or lower HRs. ### Competing Interest Statement In the 36 months prior to the initial submission, Dr. Vorland has received honoraria from The Obesity Society and The Alliance for Potato Research and Education. In the 36 months prior to the initial submission, Dr. Allison has received personal payments or promises for same from: Amin Talati Wasserman for KSF Acquisition Corp (Glanbia); Clark Hill PLC; General Mills; Kaleido Biosciences; Law Offices of Ronald Marron; Medpace/Gelesis; Novo Nordisk Fonden; Sports Research Corp.; USDA; and Zero Longevity Science (as stock options). Donations to a foundation have been made on his behalf by the Northarvest Bean Growers Association. The institution of Dr. Vorland, Ms. Henschel, Mr. Serrano, Ms. Dickinson, and Dr. Allison, Indiana University, and the Indiana University Foundation have received funds or donations to support their research or educational activities from: Alfred P. Sloan Foundation; Alliance for Potato Research and Education; American Egg Board; Arnold Ventures; Eli Lilly and Company; Mars, Inc.; National Cattlemen's Beef Association; Pfizer, Inc.; National Pork Board; USDA; Soleno Therapeutics; WW (formerly Weight Watchers); and numerous other for-profit and non-profit organizations to support the work of the School of Public Health and the university more broadly. Dr. O'Connor's research is funded by internal funds at the Agricultural Research Service, USDA and the National Cancer Institute, NIH as well as external funds from the National Institute of Agricultural, USDA and the Beef Checkoff. Dr. O'Connor also served unpaid on the National Pork Board - Real Pork Research Advisory 2nd Advisory Council. In the past 36 months, Dr. Brown has received travel expenses from Alliance for Potato Research and Education, International Food Information Council, and Soy Nutrition Institute Global; speaking honoraria from Alliance for Potato Research and Education, Calorie Control Council, Eastern North American Region of the International Biometric Society, International Food Information Council Foundation, Potatoes USA, Purchaser Business Group on Health, The Obesity Society, and University of Arkansas for Medical Sciences; consulting payments from National Cattlemen's Beef Association, and Soy Nutrition Institute Global; and grants through his institution from Alliance for Potato Research & Education, American Egg Board, National Cattlemen's Beef Association, NIH/NHLBI, NIH/NIDDK, NIH/NIGMS, and NSF/NIH. He has been involved in research for which his institution or colleagues have received grants or contracts from ACRI, Alliance for Potato Research & Education, Gordon and Betty Moore Foundation, Hass Avocado Board, Indiana CTSI, NIH/NCATS, NIH/NCI, NIH/NIA, NIH/NIGMS, NIH/NLM, and UAMS. His wife is employed by Reckitt. Other authors report no disclosures in the last 36 months prior to the initial submission. ### Funding Statement Funded by the Beef Checkoff. Supported in part by NIH grants R25DK099080, R25HL124208, and R25GM141507. The assertions expressed are those of the authors and not necessarily those of the NIH, USDA, or any other organization. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: IRB of Indiana University gave ethical approval for this work (#11227) I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Researchers who wish to reproduce our analyses can submit a project proposal to the REGARDS team. Code to reproduce our analyses is publicly available: https://osf.io/sy96k/