A Responsive Mixed-Methods Protocol for Evaluating a Multi-site Agricultural Pilot Program: The Veterans Affairs Farming and Recovery Mental Health Services (VA FARMS) Program (Preprint)

BACKGROUND Veterans Affairs Farming and Recovery Mental Health Services (VA FARMS) is an innovative pilot program that will provide supportive resources for veterans with interests in agricultural vocation. Implemented at 10 pilot sites, VA FARMS will provide mental health services and resources for veterans while simultaneously supporting training in gardening and agriculture. As each pilot site project has unique goals, outreach strategies, and implementation efforts based on the local environment and veteran population, evaluating the pilot program provides a unique challenge for evaluators. This paper describes the protocol which will be used to evaluate VA FARMS, which has been specifically designed to allow for site variation by providing both site-specific and cross-site understandings of site implementation processes and outcomes. OBJECTIVE The objectives of this paper are to (1) describe the protocol that will be used for evaluating an innovative Department of Veterans Affairs (VA) agriculturally based, mental health, and employment pilot program which serves veterans at ten pilot sites across the VHA enterprise and (2) provide guidance to other evaluators who are assessing innovative programs. METHODS This evaluation uses the Context, Inputs, Process, Product (CIPP) model, which is designed to evaluate a program’s content and implementation strategies to identify strengths and areas for improvement, enhancing program effectiveness and planning efforts. Data collection for the evaluation will use a concurrent mixed methods approach, including both quantitative and qualitative methodologies. Quantitative data collection will take place using Quarterly Program Surveys, as well as three Individual Veteran Participant Surveys administered at the veteran’s entrance and exit of the pilot program and 3 months later. Quantitative data will provide baseline descriptive statistics and follow-up statistics on veteran health care utilization, health care status, and agriculture employment status. Qualitative data collection will include participant observation at each pilot site and interviews with participants, staff, and community stakeholders. Qualitative data will include insights about pilot program implementation processes, veterans’ experiences, and short-term participation outcomes. RESULTS Evaluation efforts began in December 2018 and are ongoing. Between October 2018 and September 2020, a total of 494 veterans had enrolled in VA FARMS, and 1,326 veterans were reached through program activities like demonstrations, informational presentations, and town-hall discussions. In addition, a total of 1,623 community members and 655 VA employees were similarly reached by VA FARMS programming during that time. Data was collected between October 2018 and September 2020 in the form of 336 veteran surveys, 30 veteran interviews, 27 staff interviews, and 11 community partner interviews. Data analysis is expected to be completed by October 2022. CONCLUSIONS This evaluation protocol will provide guidance to other evaluators who are assessing innovative programs. In its application to the VA FARMS pilot, the evaluation aims adds to existing literature on nature-based therapies (NBTs) and the rehabilitation outcomes of agricultural training programs for veterans. When evaluation is complete in late 2022, results will provide programmatic insights on the implementation of pilot programs and needed improvements and modifications for the future expansion of VA FARMS and other veteran-focused agricultural programs.