We used a qualitative case study approach to examine PC CDS measurement within the real-world context of funded research projects and their investigators, allowing for in-depth exploration of processes, challenges, and contextual factors [9]. We used four methods for this case study approach: (1) systematic review of relevant projects funded by the AHRQ Digital Healthcare Research Program, (2) document review of project artifacts, (3) semistructured key informant interviews, and (4) expert review via a technical expert panel (TEP).

We conducted a systematic review of PC CDS projects funded by the AHRQ Digital Healthcare Research Program [10,11]. This program funds research across the United States to advance the evidence base for digital health care and represents one of the most comprehensive federal portfolios dedicated to advancing PC CDS in real-world settings.

The inclusion criteria included projects that (1) were completed by 2024; (2) had at least one published artifact, such as a final report, manuscript, or presentation; (3) involved the use of patient-contributed data, such as PGHD, PROs, patient preferences, and/or HRSN data by a PC CDS technology [12]; and (4) included an evaluation component in the project. Projects were excluded if they did not meet one or more of the inclusion criteria.

Using project descriptions available on the AHRQ’s Digital Healthcare Research website, two independent researchers (authors CZ and NG) screened titles and abstracts of 67 projects, ultimately excluding 41 projects that did not meet the inclusion criteria. This was followed by a full-text review of publicly available documents for the remaining projects (n=26) that were linked on the website, including final reports, publications, and public webinars. Discrepancies at each review stage were resolved through consensus discussions, with arbitration by a third independent reviewer (author PD), when required. A total of 20 projects were ultimately selected for inclusion in the review, with a total of 40 associated publications. The selected projects provide a range of perspectives across PC CDS technologies, populations, and medical conditions, supporting applicability of findings to other health systems or settings.

Two researchers (CZ and NG) independently performed data extraction using a standardized data abstraction form in Microsoft Excel that collected information on the following criteria: project characteristics (eg, title, years funded, principal investigator), PC CDS project phase as per the PC CDS Lifecycle Framework (eg, design and development, implementation and evaluation) [13], type of patient-contributed data collected (eg, PGHD, PROs), description of data collected (eg, step count data from a smartwatch), population addressed with the PC CDS intervention, type of technology used, list of measures used, and data collection methods. The abstraction form can be found in Multimedia Appendix 1.

To identify and extract the list of measures from included studies, the research team used ChatGPT-3.5 to conduct an initial review of each study. A large language model was used to support the initial extraction of performance measures because identifying and cataloging measures required detailed review of a large body of literature (40 studies), including studies in which measures were not consistently labeled or explicitly described. The tool was used as an aid to improve the efficiency and consistency of the extraction process and to facilitate the identification of candidate measures for reviewer assessment, but it was not used as a substitute for human review. Between October and November 2023, research team members developed study-specific prompts that were iteratively refined, as needed, to improve completeness and specificity of extracted information, particularly for qualitative studies in which measures were not explicitly defined (see Multimedia Appendix 2 for the list of baseline prompts). All artificial intelligence (AI)–generated extractions underwent independent review by a research team member (CZ or NG), who compared the output against the full-text paper and corrected or supplemented the extracted information, as needed. Particular attention was paid to verifying accuracy and describing measures at the most specific level possible, including the names of instruments, scales, and PRO questionnaires, when reported. Following quality assurance review, the verified measures were incorporated into the Microsoft Excel abstraction form.

We purposively selected 10 projects for key informant interviews with the project principal investigators to gather their perspectives on PC CDS performance measures and measurement-related gaps and challenges. In selecting projects, we aimed to capture diversity in the type of PC CDS technology and patient-contributed data, performance measures evaluated, use case, and phase of the project (eg, design, development, implementation). We also considered recommendations from the AHRQ regarding projects with rich implementation experience that could provide particularly informative perspectives. Of the 10 invited principal investigators, 9 agreed to participate and completed an interview. The high participation rate may have been influenced by knowledge that this study was funded by the AHRQ. To mitigate response bias, interviewers emphasized that participation is voluntary and that responses would be reported at an aggregate level and would not be attributed to individual participants.

We conducted 60-minute individual interviews virtually on Zoom. Interviewers used semistructured discussion guides tailored to each project (see Multimedia Appendix 3). This ensured consistency in data collection, while allowing flexibility to explore unique project contexts and emergent themes. Discussions focused on (1) experiences selecting and using measures to assess PC CDS performance, (2) perspectives on gaps and challenges in PC CDS performance measurement, and (3) perspectives on opportunities to improve PC CDS performance measurement. A senior team member (author PD or DFS) led the discussions, and an analyst recorded transcript-style notes and key takeaways. All interviews were audio-recorded, and all notes were reviewed by four members of the research team (CZ, NG, PD, and DFS) to ensure agreement on key takeaways.

We engaged a TEP convened by the AHRQ-funded Clinical Decision Support Innovation Collaborative (CDSiC), a multiyear project focused on PC CDS, to validate findings and refine interpretations. The TEP included seven experts in CDS, including informaticians, developers, and patient advocates. Over three virtual meetings, we presented interim results and received feedback on (1) comprehensiveness of the identified measures, including notable gaps; (2) relevance of measurement challenges and opportunities raised by key informants; and (3) potential strategies for advancing the field. All meetings were audio recorded (guided discussion questions can be found in Multimedia Appendix 4). An analyst documented detailed meeting minutes, which were refined using the audio recordings and reviewed by senior research team members (PD and DFS) for accuracy. TEP feedback was used to refine findings and inform development of the action plan for PC CDS performance measurement.

We used a sequential synthesis design in which each data collection phase informed subsequent phases of the study to refine our analysis. Specifically, findings from the document review were used to develop and refine the key informant interview guide, enabling us to explore project-specific factors. We then validated and refined our integrated findings with TEP members to support triangulation.

For the document review, two researchers (CZ and NG) independently mapped each measure to the subdomains of an existing PC CDS performance measurement framework and identified measures that did not fit well into the framework [14]. The PC CDS performance measurement framework is informed by constructs from previously established health IT and CDS evaluation frameworks and provides domains and subdomains for measurement to understand the performance and impact of PC CDS technology (Table 1). This framework was selected because it is specifically designed for PC CDS technologies, which require consideration of the full PC CDS lifecycle, patients’ needs and preferences, and outcomes at multiple levels (individual, population, and health system). Several members of the study team (PD, DFS, CZ, and NG) contributed to the framework’s development, and additional testing and independent external validation are needed to further evaluate and refine the framework.

Any discrepancies were resolved through discussion with a third researcher (PD) until a consensus was reached. We used descriptive statistics (eg, frequencies and counts) to summarize measure types across projects to support identification of common areas of measurement and measurement gaps. Additionally, we reviewed each paper’s discussion and limitations sections to identify measurement-related challenges. Using the web-based brainstorming platform Lucidchart [15], which enables real-time collaboration with visual diagramming, two research team members (CZ and NG) abstracted each challenge onto a virtual sticky note and grouped them into broader themes, which were reviewed and refined with a senior researcher (PD).

For the key informant interviews and TEP meetings, we conducted rapid thematic analysis of transcripts and TEP meeting minutes. The lead notetaker reviewed the transcripts and meeting minutes and prepared structured summaries of each interview and TEP meeting that captured higher level themes related to measurement gaps, challenges, and recommendations. After all interview and TEP meeting-specific themes were finalized, two research team members (CZ and NG) grouped them into broader cross-cutting themes. We conducted regular debriefings with the broader study team, including researchers not involved in data collection, to refine emerging themes and reconcile discrepancies through discussion to reach consensus. We conducted a qualitative thematic synthesis to integrate findings across the document review, interviews, and TEP meetings. Themes from the interviews and TEP meetings were compared with those from the document review to validate interpretations, identify gaps and challenges, and enhance the preliminary synthesis. TEP themes also informed the action plan for PC CDS performance measurement.

The NORC at the University of Chicago Institutional Review Board (FWA00000142) reviewed all methods and deemed this work not human subjects research. Key informant interview participants were informed about the purpose of the study, the voluntary nature of participation, and the planned use of study data. Verbal informed consent was obtained from all participants prior to conducting interviews. Interview and TEP meeting recordings and transcripts were stored in a secure NORC environment with access restricted to authorized research team members. To protect confidentiality, participants were anonymized and findings synthesized at the aggregate level. No compensation was provided to participants for their involvement in the study.

Figure 1 shows the flowchart for identifying projects via the AHRQ’s Digital Healthcare Research Program. Details of the 20 projects [16-35] included in the review can be found in Multimedia Appendix 5. The projects involved PC CDS technologies in stages of (1) design, (2) development, (3) implementation, and (4) evaluation (Figure 2). The design and development phases involve identification of user needs and requirements, preliminary designs, rapid prototyping with user input, user testing and process, and technology changes prior to implementation [36,37]. The implementation phase involves the deployment of the PC CDS into clinical workflows [36-38] or patients’ daily activities, and the evaluation phase is the process of measuring whether the tool achieves its defined objectives [36,38,39]. Most projects (n=12, 60%) covered all phases of the PC CDS lifecycle, while some focused only on implementation and evaluation (n=7, 35%) or design and development (n=1, 5%).

The PC CDS technologies addressed various medical conditions and/or HRSN, with the most common being chronic conditions (n=11, 55%), such as arthritis, asthma, cancer, diabetes, or HIV. Other focus areas included health behavior/education, such as diet, medication adherence, physical activity, or preconception care (n=4, 20%); mental health or neurologic conditions, such as attention-deficit hyperactivity disorder, depression, or dementia (n=3, 15%); social needs, such as changes in life circumstances and behaviors or intimate partner violence (n=2, 10%); and acute conditions, such as dental pain or gastrointestinal bleeding (n=2, 10%).

The projects used different types of patient-contributed data to inform PC CDS interventions, with the most common being PROs for specific conditions, such as depression, quality of life, and pain (n=13, 65%). Other types included PGHD collected by devices, such as blood pressure and step count (n=4, 20%); HRSN, such as job loss (n=3, 15%); and patient preferences, such as treatment options (n=1, 5%). In terms of the technology used, the projects frequently leveraged mobile apps/smartwatches (n=9, 45%) and electronic health records (EHRs; n=7, 35%), while a few used machine learning (ML) or AI (n=3, 15%), web-based tools (n=3, 15%), or text messaging (n=2, 10%). Patients were the most common end users of the technologies (n=18, 90%), although many were both patient- and clinician-facing technologies (n=13, 65%), with 2 (10%) being developed for caregivers.

When the type of patient-contributed data and type of technology were examined together, we found that projects collecting and using PROs leveraged all four types of technology (mobile apps/smartwatches, EHRs, AI/ML, and web-based tools), as shown in Figure 3. All PGHD were collected and shared using mobile apps/smartwatches or text messaging directed to the patient. All projects using HRSN data did so by using EHRs for collection and display. Only 1 (5%) project collected patient preference data using the EHR.

Here, we present common measurement areas in PC CDS projects, identified through our document review, across both the design and development and the implementation and evaluation phases. Measurement in the design and development phases focused on assessing end-user needs and preferences. Measurement in the implementation and evaluation phases extended beyond usability to assess impact and effectiveness.

To explore gaps in PC CDS measurement, we asked key informants to elaborate on those identified in the document review and to provide additional insights based on their research and practice. Here, we describe several key gaps and challenges in measurement across PC CDS phases:

TEP members provided guidance and feedback on preliminary findings during the beginning, middle, and end of the study. Overall, the panel responded positively to the findings and action plan, affirming their relevance, while pointing to key opportunities to strengthen measurement approaches, standardization, and evidence generation in the PC CDS field.

Engagement emerged as a central theme, with members underscoring both its importance and complexity. They observed that patient engagement varies significantly depending on clinical context, with patients experiencing acute or high-salience conditions (eg, pregnancy or cancer) often demonstrating higher engagement than those managing chronic conditions. Members also discussed the need for better standardization of engagement measures and distinguishing between process measures (eg, onboarding steps, such as downloading an app or logging in) and outcome measures (eg, whether patients take an intended action). However, attributing outcomes directly to PC CDS remained challenging due to confounding factors, reinforcing the need for rigorous study designs (eg, A/B testing, control groups) and improved statistical approaches. They also highlighted the importance of comparing engagement results to baselines or benchmarks and accounting for organizational scale. For example, what may appear to be modest engagement rates could still translate into meaningful improvements in reach and cost savings in large health systems.

Members also provided insights into gaps in current PC CDS measurement. Similar to key informants, TEP members suggested that limited attention to domains such as cost and technical performance may reflect the early-stage nature of many PC CDS projects, which are often focused on feasibility rather than long-term performance or scalability. At the same time, members found it notable that technical performance is not more consistently measured, given its importance for patient trust and system usability.

Our review sought to identify measures currently used to evaluate PC CDS technologies in federally funded digital health projects and to characterize gaps in existing measurement approaches as they are increasingly integrated into clinical care. We found that usability, acceptability, patient health outcomes, patient engagement, and clinician performance were the most commonly measured domains in the selected PC CDS studies, while relatively little attention was given to technology safety, timeliness, and cost. This is notable because these domains may substantially influence the sustainability, scalability, and real-world value of PC CDS technologies [57,58]. The absence of these measures suggests that current measurement approaches may provide an incomplete picture of PC CDS performance, particularly as tools move from research settings into routine clinical practice.

We also identified persistent challenges in measuring some of the commonly assessed domains that underscore the unique complexities of evaluating PC CDS technologies that interact directly with patients. Projects included in this review found that existing usability instruments are often not well suited for patient populations with limited digital health literacy, a concern that has been raised in digital health research more broadly [59] but may be particularly consequential for PC CDS because these technologies depend on patients’ ability to interpret and act on health information outside of traditional clinical encounters. Additionally, although the prior literature has highlighted the importance of patient engagement for the success of digital health interventions [60], our review identified that measuring patient engagement for PC CDS specifically is complicated by the use of third-party applications that support these technologies and health system privacy and security concerns that restrict their collection and use of data.

Other challenges identified in this review mirror longstanding issues in health information technology evaluation. Assessment of patient health outcomes is frequently limited by the short duration and scale of many studies, while measuring clinician use of technologies remains difficult because relevant data are often embedded within complex EHR systems. Although these barriers are not unique to PC CDS, our review demonstrates how they continue to inhibit measurement and therefore our understanding of how technologies can improve care.

Although the projects in this review may not be applicable across all PC CDS contexts, collectively, these findings indicate six important next steps for future development of PC CDS performance measurement as the field continues to mature:

This study has some limitations. The document review was restricted to a single funding program, the AHRQ’s Digital Healthcare Research Program, that aims to advance digital health care technologies that address the evolving needs of patients, clinicians, and health systems. This limits the generalizability of findings to other funding mechanisms or research settings. Additionally, the key informant interviews were conducted with a purposively selected subset of the included projects to capture a range of PC CDS technologies, patient-contributed data, and evaluation experiences. As such, the interview findings may not reflect the perspectives of all projects within the portfolio or the broader PC CDS field. Third, participation in interviews was voluntary, introducing the potential for selection bias in interview participation, as principal investigators who agreed to be interviewed may represent a particular perspective on their own work. Finally, interview responses may have been impacted by self-report biases, as investigators may have been reluctant to acknowledge challenges or shortcomings within their own projects and may attribute limitations to external factors instead.

This study identified PC CDS performance measurement areas that are well defined and represented in 20 real-world PC CDS projects, as well as gaps that limit our understanding of how these technologies perform in practice. Although we only reviewed a small subset of PC CDS projects in the formative research stage, the findings suggest that there is not yet an established, comprehensive approach to evaluating PC CDS technologies across different patient populations, settings, and stages of implementation. To advance the development and use of PC CDS, future efforts should (1) measure technical performance after implementation to understand how tools function in real-world use; (2) evaluate interventions in low-resource health care settings to account for varied infrastructure and capacity; (3) conduct more cost assessments to inform feasibility and scalability; (4) develop practical approaches to managing health systems’ privacy and security concerns with storing individual data on external apps and servers; (5) refine and develop standardized patient engagement measures to assess who is engaging with PC CDS technologies, how well they are engaging, and how engagement impacts outcomes; and (6) adapt measurement tools to populations with limited English proficiency or digital and/or health literacy. Addressing these areas can strengthen the evidence base for PC CDS and support development of technologies that promote patient-centered care.