A Sociotechnical Framework to Assess Patient-Facing eHealth Tools: Results of a Modified Delphi Process

Abstract Background : Novel healthcare technologies have a promising potential. However, among the thousands of eHealth tools available, the vast majority do not get past pilot phases because they cannot prove value, and only a few have been systematically assessed or evaluated. Although multiple eHealth assessment frameworks and initiatives have been developed over the last decade, showing a lack of standardization in this field, these efforts face multiple challenges: lack of validation, assessor diversity, contextuality, subjective measures, and lack of practicability. Objective : This study aimed to address some of the identified challenges by validating an initial list of assessment criteria that resulted from a systematic review conducted by the research team through a diversified expert panel. Expert consensus helped define which criteria were must-have criteria, which criteria were not as important, and whether there were any criteria that were missing and should be added to the validated framework. Additionally, discussions with experts went beyond the final list of criteria to address their diverse perspectives on how to make the assessment instrument as accessible and usable as possible. Methods : We conducted a two-round modified Delphi process with in-between rounds of interviews to validate and refine an initial list of 55 assessment criteria that we synthesized based on a systematic literature review of pre-existing assessment frameworks. The expert panel (N=57) included participants from 18 countries and 9 stakeholder groups: eHealth experts, clinicians, patients and patient advocates, researchers, pharma executives, insurance and reimbursement experts, regulatory experts, investors, and eHealth technology providers. Consensus was defined a priori as at least 75% agreement. Two rounds of voting were conducted, electronically. Prior to Round 2, one-to-one semi-structured interviews were conducted to explore the perspectives of diverse stakeholders on some key challenges and directional decisions regarding the proposed assessment instrument. Results : A consensus was reached on 46 assessment criteria that were classified into core and contextual criteria. The 36 core criteria focus on evaluating the eHealth tool itself regardless of its context and were grouped into nine clusters: technical aspects, clinical utility and safety, usability and human centricity, functionality, content, data management, endorsement, maintenance, and developer. The 10 contextual criteria focus on evaluating the factors that vary depending on the context the tool is being evaluated for and were grouped into seven clusters: data-protection compliance, safety regulatory compliance, interoperability and data integration, cultural requirements, affordability, cost-benefit, and implementability. Conclusions : The classification of assessment criteria into core and contextual helps us assess not only the quality of an isolated tool, but also its potential fit in a specific healthcare setting. The criteria subscales may be particularly relevant when determining the strengths and weaknesses of the tool being evaluated. This granularity enables different stakeholders to make informed decisions about which tools to consider according to their specific needs and priorities.