The integration of digital health solutions (DHSs) into health care has become an area of extensive research [1,2]. For our study, we adopted the European Commission’s definition of DHSs, which describes them as “tools and services that use information and communication technologies to improve prevention, diagnosis, treatment, monitoring and management of health-related issues and to monitor and manage lifestyle-habits that impact health” [3]. The many ways in which DHSs can transform health care have been shown in various studies, such as achieving cost and time savings [4], improving medical safety and treatment adherence [5], enhancing service quality [6], and ameliorating health outcomes [7].

Despite these benefits, the adoption of DHSs in health care is slow [8]. This is even more pronounced in outpatient settings compared to inpatient settings [9,10]. The key obstacles include required workflow adjustments, high costs for installation or maintenance, and unclear or limited effectiveness [11,12].

These challenges, especially the considerations between effectiveness and costs, underscore the importance of reliably measuring the impact of DHSs. Reliable measurement can support the further adoption of DHSs, that is, by helping decision makers compare the effects of different DHSs, select the most suitable DHS for their organization, and monitor the progress of deployments [12-14].

Moreover, as the landscape of DHSs grows increasingly complex with more and more digital use cases, tools, and providers [15], it is becoming more difficult for decision makers to navigate this landscape. Therefore, robust evaluation frameworks become more important for their support [16]. Recent literature reveals that the instruments currently used to assess the implementation of DHSs often lack robust psychometric properties, leading to a lack of consensus on preferred assessment methodologies [17-19]. Existing validated instruments tend to focus on specific use cases or diseases, which limits their utility for comparison across multiple DHSs and health care settings [20-22]. This variation in quality and scope of reported effects complicates the task for decision makers attempting to compare, select, and implement DHSs.

To address the limitations of currently applied instruments, it is essential to develop and establish comprehensive instruments. Existing studies have used a range of broadly applicable instruments, for example, measuring costs, resource use, patient satisfaction, and quality of life [20]. For example, the Mobile Healthcare App Database uses the Mobile Application Rating Scale to increase transparency regarding the quality of content and data security of >1300 DHSs [23,24].

Although service quality is a well-established and extensively studied outcome in various sectors, including health care—with numerous validated instruments available [25]—to our knowledge, established instruments for health care service quality (HCSQ) evaluation have not yet been used to assess the effects of DHSs. Recognizing the necessity for a versatile outcome measure, we considered whether HCSQ might be an effective metric to assess the effects of DHSs in a variety of health care settings. Therefore, this study aimed to address the lack of broadly applicable and reliable instruments by identifying, adapting, or developing a questionnaire that can measure the service quality of DHSs across different settings leveraging existing instruments. The broad adoption of such a tool would be particularly valuable for decision makers when considering the selection and implementation of DHSs.

This study used a sequential approach (Figure 1) following predefined construction criteria to create a comprehensive questionnaire that can measure HCSQ in settings with different DHSs, across therapeutic areas, and from both patient and health care practitioner (HCP) perspectives for the assessment of the effects of DHS deployment on HCSQ.

We selected Health Care Service Quality (HEALTHQUAL) [26]—a validated, health care–specific service quality instrument based on the well-established service quality model Service Quality (SERVQUAL) [27]—for further adaptation to the context of this study. As HEALTHQUAL has not previously been used in digital health settings, we conducted a rapid review adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [28], as well as recommendations from Tricco et al [29] and King et al [30]. This review helped identify existing instruments used to assess the postdeployment effects of DHSs. The Outpatient Experience Questionnaire (OPEQ) was selected for the subsequent steps [31,32].

Neither the original HEALTHQUAL nor OPEQ have been validated for use with HCPs. To address this limitation, we conducted semistructured qualitative interviews with 19 HCPs, following the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist [33]. This process validated the relevance of the included dimensions for HCPs and led to the addition of a dimension for patient satisfaction, which was derived from the Patient Satisfaction Questionnaire-Short Form (PSQ-18) after an additional literature review on patient satisfaction measurement instruments [34].

Consequently, we derived our suggested set of dimensions and items, hereafter referred to as Digital Healthcare Service Quality (DigiHEALTHQUAL).

To identify, adapt, or develop a questionnaire, we defined the following five criteria (Figure 1):

One well-established tool in the area of health service is SERVQUAL [27], which is widely used to measure service quality across different sectors [38]. SERVQUAL has often been adapted to measure HCSQ in various health care settings [25,39]. This led to the development of several health care–specific variations of SERVQUAL. The 3 frequently used variations are called HEALTHQUAL and were developed by Miranda et al [40], Lee [26], and Mosadeghrad and Sokhanvar [41]. Despite evidence that DHSs impact HCSQ [1,6], none of these HEALTHQUAL instruments have been used to assess these effects. Therefore, this study aimed to fill this gap by adapting HEALTHQUAL by Lee [26] to evaluate the impact of DHSs on HCSQ.

For this study, we used HEALTHQUAL by Lee [26] because of its comprehensive design approach, which incorporated input from experts and patients as well as international health care service accreditation systems. HEALTHQUAL by Lee [26] closely aligns with the original SERVQUAL framework and adds a dimension for health outcomes, making it a robust tool for assessing HCSQ [26]. In addition, HEALTHQUAL by Lee [26] has been widely applied internationally in both inpatient and outpatient settings (eg, South Korea, India, Syria, and Ghana) [42-47]. The instrument consists of 5 dimensions with a total of 32 items. These dimensions are empathy, tangibles, safety, efficiency, and the degree of improvements of care services.

We conducted qualitative, semistructured interviews with 19 HCPs. These interviews followed the COREQ checklist [33]. More details can be found in Multimedia Appendix 3.

We designed an interview guide to validate findings from rapid review stage 1 and to explore additional HCSQ dimensions relevant to HCPs. More details are provided in Multimedia Appendix 4. The interviews focused on two domains: (1) implementation and current experience with DHSs and (2) evaluation and further exploration of HCSQ dimensions.

Interviewees were recruited through targeted outreach to both contacts within the researchers’ network as well as to HCPs with relevant profiles and publicly available contact information. The following profiles were included: (1) working as an HCP [36], (2) working in outpatient care [37], and (3) considering the implementation or already using at least 1 of the selected DHS use cases. We ensured a diverse sample across specialties, roles, demographics, and countries.

The interviews were conducted with each interviewee individually over the phone. Each session was audio recorded and transcribed verbatim. Recruitment concluded after 19 interviews, upon reaching data saturation. The average duration of each interview was 29 (SD 8) minutes.

We performed coding and qualitative analysis using MAXQDA software (Verbi software) [61]. Our content analysis approach combined deductive and inductive methods [62]:

On the basis of these analyses, we quantitatively scored the relevance of each literature-derived dimension.

The study protocol was reviewed and approved by the Ethics Committee of Witten/Herdecke University (S-113/2023). All interviewees received written information outlining the study’s objectives, procedures, and data handling practices and provided written informed consent before participation. Participation was entirely voluntary, and no incentives were offered. Audio recordings were deleted following verbatim transcription. During transcription and subsequent analysis, all personally identifiable information was removed. The interview data were factually anonymized, and results were presented only in aggregated form, making individual identification possible only with disproportionate effort. All data were stored on secure, password-protected systems.

Our objective was to identify which instruments are currently used to measure the effects of DHSs. Our query yielded 9043 publications. To focus on the most current publications, we limited our inclusion to articles in English or German and published within the last 5 years, up to 2023. After removing duplicates, 4957 articles remained for screening, which resulted in 98 potentially relevant articles. Of these, 62 articles were accessible and underwent full-text screening, ultimately leading to the inclusion of 40 publications (Figure 2).

In the 40 eligible studies, we identified a total of 126 reported outcomes. We applied the eligibility criteria used during literature screening to these outcomes analogously, resulting in 85 quantitative, nonhealth-related outcomes applied in outpatient settings measuring the postdeployment effects of selected digital use cases. Many of these were based on self-constructed instruments without reporting validity or reliability characteristics. Only 23 were based on instruments that have either been validated or have been used in previous publications [32,63-76].

In total, we identified 18 distinct instruments (Table 1). We assessed the instruments to ensure adherence to our predefined construction criteria, especially for applicability across digital use cases and perspectives. None of the instruments fulfilled both criteria; instead, 3 instruments assessed both the patient’s and HCPs’ perspectives but were use case–specific [63,67,69,72], and 9 instruments were not use case–specific but did not assess different perspectives [32,64-68,70,73,74,76].

For further instrument development, we hypothesized that adjusting an existing instrument from the patient perspective to the HCP perspective would require fewer changes to the original instrument than making a use case–specific instrument more generalizable.

To select an instrument, we focused on those that had been applied multiple times, which is the case for the Consumer Assessment of Health Care Providers and Systems (CAHPS) [64,66] and OPEQ [32,65]. We mapped their dimensions against the HCSQ dimensions outlined by Lee [26]. The comparison revealed that both CAHPS and OPEQ cover dimensions similar to those of HEALTHQUAL (Figure 3 [26]). Notably, OPEQ covers a slightly broader range of HCSQ dimensions compared to HEALTHQUAL. Therefore, we selected OPEQ for further use, merged mapping dimensions with HEALTHQUAL, and added accessibility (merging clinic access and previsit communication) and information as additional HCSQ dimensions.

Given that HEALTHQUAL and OPEQ have not been previously used with HCPs, we conducted interviews with HCPs with two objectives: (1) to confirm existing and derive new dimensions for the evaluation of HCSQ from the perspective of HCPs and (2) to understand the importance of each dimension for the overall HCSQ.

We interviewed 19 HCPs with diverse backgrounds (Figure 4). We derived a total of 11 HCSQ dimensions from the interviews, with a broad consensus for four of them (mentioned by at least 12/19, 60% of interviewees):

Other less frequently mentioned dimensions included providing information, the atmosphere in the practice, available medical offerings, and improved health conditions. These dimensions match those in HEALTHQUAL and OPEQ. Notably, 4 (21%) out of 19 interviewees explicitly mentioned patient satisfaction as a separate dimension of HCSQ not represented by the dimensions of HEALTHQUAL and OPEQ. The 2 dimensions not represented by the dimensions of HEALTHQUAL and OPEQ were mentioned but with low frequency. Two interviewees mentioned integrated care and the continuous effort of the practice to improve the service.

Furthermore, interviewees rated the relevance of the HEALTHQUAL and OPEQ dimensions (Figure 5). We found that there was a consensus regarding the most relevant HCSQ dimension (safety) but more variation regarding the remaining dimensions. However, all dimensions received a rating of at least 7 on a scale of 10, so we considered all of them as relevant.

These findings confirmed the relevance of the selected HCSQ dimensions for HCPs. However, patient satisfaction was mentioned by >20% (4/19) of interviewees as an additional relevant dimension. Consequently, we conducted another rapid review to identify instruments measuring patient satisfaction in the literature to be added to our questionnaire.

The objective was to identify the instruments currently used to measure patient satisfaction. We decided to narrow down the scope of this review further to primary health care settings because the 4 mentions of patient satisfaction were made by HCPs working in primary health care. Our initial search resulted in 1573 publications. To focus on recent research, we included only articles written in English or German that were published within the last 5 years, up to 2023, reducing the dataset to 493 articles. Further screening narrowed the selection to 136 potentially relevant articles. Of these, 105 were accessible and subjected to full-text review, leading to the selection of 35 publications (Figure 6).

In the 35 eligible studies, 29 different instruments related to patient satisfaction were used or mentioned (Tables 2 and 3). To select the most suitable instrument to add to our questionnaire, we evaluated the following criteria: (1) the frequency of use, (2) the internal consistency (ie, Cronbach α), and (3) the feasibility for integration (ie, number of items).

A total of 6 instruments (accounting for different variations or versions of an instrument as 1) were used multiple times. All of them demonstrated overall acceptable Cronbach α values. However, CAHPS and Primary Care Assessment Tool were reported to have Cronbach α values <0.7. The Patient Satisfaction Questionnaires and the General Practice Assessment Questionnaire contained the shortest instruments, with 18 and 19 items, respectively. On the basis of these criteria, we selected the most recent version of the Patient Satisfaction Questionnaires, the PSQ-18, for inclusion in our questionnaire based on its widespread use in the literature, reliability, and feasibility.

We developed DigiHEALTHQUAL by synthesizing 3 validated instruments—HEALTHQUAL, OPEQ, and PSQ-18—guided by insights from our rapid reviews and interviews in the previous steps. HEALTHQUAL measures HCSQ via 32 items across 5 dimensions (empathy, efficiency, safety, tangible, and degree of improvements of care services), with responses rated on a 5-point scale ranging from “very bad” to “very good” [26]. The OPEQ assesses outpatient experiences through 24 items in 6 dimensions (clinic access, communication, hospital standards, organization, previsit communication, and quality of information), using a 10-point scale with anchoring phrases for each end of the scale [31,32]. The PSQ-18 evaluates patient satisfaction via 18 items across 7 domains (accessibility and convenience, communication, financial aspects, general satisfaction, interpersonal manner, technical quality, and time spent with doctors) on a 5-point Likert-like scale ranging from “strongly agree” to “strongly disagree” [34].

The 3 instruments were compared item by item to remove duplicates and ensure contextual applicability by adapting or removing items as needed (Multimedia Appendix 5). Of the initially longlisted 74 items, 13 were adjusted to better fit the focus of our study, 24 were removed because of redundancy or to account for differences between national health care systems, and 1 new item was added because adjusting the existing question to fit the focus of this study was not feasible (ie, data security considerations).

Subsequently, the remaining dimensions of the 3 original instruments were merged based on content overlap where feasible. The resulting DigiHEALTHQUAL questionnaire consists of 51 items across 8 dimensions: accessibility, efficiency, empathy, general satisfaction, degree of improvements of care services, information, safety, and tangibles.

This current questionnaire draft is designed to use a response scale similar to that of the PSQ-18, a 6-point Likert scale ranging from “completely disagree” to “completely agree,” allowing calculation of a satisfaction score for each subdimension. This scale has proven effective in practice [12]. Consequently, all items were converted accordingly.

This study reveals key findings regarding the measurement of the effects of DHSs on HCSQ. First, there is a notable lack of consensus on standardized instruments to measure the effects, especially for nonhealth-related outcomes. We hypothesize that this lack of agreement is a reflection of the novelty and wide variety of available DHSs, which consequently makes it challenging for decision makers to compare the effects of different DHSs across diverse contexts.

In contrast, there is greater agreement on the instruments used to measure patient satisfaction, indicating that these tools are well established and widely accepted. This consensus provides a solid foundation for evaluating patient satisfaction in the context of DHSs, offering a reliable starting point for further research.

Finally, instruments typically used to measure HCSQ have not yet been used to assess the impact of DHSs. This gap underscores the need to adapt existing HCSQ instruments for digital use cases. Interestingly, some of the instruments identified in this study include dimensions relevant to HCSQ. This indicates that some level of assessment of HCSQ elements is already taking place, but it is not being explicitly reported as such.

Previous research has highlighted a lack of consensus on instruments for measuring implementation research outcomes in health care, including DHS deployments, primarily because of insufficient psychometric properties [17-19]. Our study confirms this general observation and provides additional insights specific to DHS implementation research outcome measurement. We found that the lack of standardized instruments is particularly pronounced for nonhealth-related outcomes in DHS implementation studies. Many of these studies used self-constructed instruments that neither referenced previous work nor reported validity or reliability properties. This variability and lack of rigor make it challenging to compare findings across different studies and contexts. Consequently, our findings underscore the need to validate new or existing instruments for nonhealth-related outcomes to enhance the reliability and comparability of DHS implementation studies and generate a consensus within the research community related to the applicable instruments.

In our study, we identified instruments to measure the effects of DHSs and several instruments to measure HCSQ. However, none of the currently used instruments explicitly assess the intersection of these 2 areas: how HCSQ is influenced by DHS deployment. Recognizing this gap, we followed the recommendations of Boateng et al [117] by choosing to adapt and combine existing instruments rather than create a questionnaire de novo. This approach also aligns with the experimental practice of modifying established instruments such as SERVQUAL to fit specific health care contexts, although it has not yet been widely applied to DHSs [118]. By following this adaptation strategy, we ensure the relevance and applicability of our instruments while maintaining the validity, reliability, and comparability of the results by leveraging well-established and previously validated instruments.

Our HCP interviews revealed the recommendation to add patient satisfaction as an additional dimension to the set of HCSQ dimensions derived from HEALTHQUAL and OPEQ. This recommendation is anecdotally already supported by our rapid review stage 1, for example, where the CAHPS instrument included a question on overall satisfaction. The literature consistently described that there is a correlation between HCSQ and patient satisfaction or happiness in health care settings [119-121]. Various studies identify factors influencing patient satisfaction. For example, Duc Thanh et al [120] identified key factors affecting patient satisfaction, including satisfaction with facilities, service provision, information transparency and administrative procedures, accessibility, and staff interaction and communication. Similarly, Ferreira et al [119] highlighted factors such as waiting times, information provided, cleanliness, communication with patients, and doctors’ characteristics. These factors affecting patient satisfaction align closely with the HCSQ dimensions outlined by Lee [26], underscoring the interconnectedness of the metrics patient satisfaction and HCSQ and their collective impact on health care evaluation—supporting the inclusion of patient satisfaction in DigiHEALTHQUAL.

By leveraging existing, validated instruments rather than creating new ones, we ensure comparability with previous studies. This approach not only strengthens the validity and reliability of DigiHEALTHQUAL but also facilitates meaningful comparisons and meta-analyses for future research. In addition, our study aimed to add the HCP perspective to patient-focused instruments, providing a holistic view of HCSQ. This comprehensive approach ensures that the nuances and needs of both key stakeholder groups are considered in future research, enhancing the overall applicability and relevance of DigiHEALTHQUAL. Moreover, our approach aligns with the best practices for developing and validating scales for health research, as described by Boateng et al [117], who describe 9 steps across 3 phases for the development of rigorous scales. Following their work, we completed the first phase, “item development,” comprising the 2 steps “identification of domain and item generation” and “content validity” in this study [117].

To enable broad application, we excluded dimensions from the original instruments, for example, items related to financial aspects, such as “appropriate costs” or “reasonable medical expenses,” because of varying reimbursement across national health care systems. This may result in a lack of insights into selected aspects of HCSQ. Furthermore, the interviews informing our questionnaire design were conducted exclusively with HCPs from high-income countries, which may limit the applicability of the resulting questionnaire to countries with different health care systems. However, the interviews only informed the decision to keep the HCSQ dimensions derived from the validated instruments HEALTHQUAL and OPEQ and to add patient satisfaction as an additional dimension—also derived from a validated instrument used in international studies. Therefore, we estimate that the geographic bias of the interviewees is negligible at this stage of our research. It is important to note that our resulting country-unspecific instrument may have limitations in practice, where national regulatory frameworks influence the adoption of DHSs and data use [15,122].

Generalization issues also arise, as some selected digital use cases may not impact all included HCSQ dimensions. For example, we theorize that online appointment booking platforms are rather unlikely to affect health outcomes. Furthermore, nonprioritized DHSs may require other evaluation criteria. For instance, we hypothesize that the effects of Digital Health Applications will materialize mainly in health outcomes rather than HCSQ and, therefore, necessitate specific, targeted assessment tools. This underscores the need for researchers to balance between using broadly applicable instruments that facilitate comparison across studies and specific instruments that provide detailed insights.

Within the broader context of health care, it is important to note the following: merely assessing the effects of DHSs is insufficient to accelerate their adoption. Comprehensive implementation strategies are necessary to facilitate widespread use.

Moreover, while scaling the digitalization of health care can improve health care delivery, it may also reinforce or exacerbate existing disparities. Health care access and quality are already unevenly distributed, and DHSs risk exacerbating these existing inequalities [123].

On the one hand, digital technologies can help mitigate disparities, for example, by improving access in underserved areas or personalizing care. On the other hand, they can also create new barriers for populations with limited digital literacy, lower socioeconomic status, or restricted access to technology. Studies show that the use of digital health tools correlates with sociodemographic factors such as age; income; education; and, critically, health and digital literacy [124-126].

This introduces 2 challenges. First, in the context of validating DigiHEALTHQUAL, it is important that diverse user groups are appropriately represented. Digitally literate individuals are easier to recruit through online channels, whereas individuals with limited digital access or skills may be systematically underrepresented, leading to bias in the findings. Second, beyond the scope of our work, there is a need for comprehensive planning that ensures equity of access. DHSs must be designed and implemented in a way that safeguards against creating additional barriers and addresses common user concerns such as data privacy, usability, and accessibility.

The implications for further research based on our study are threefold.

Our study underscores the importance of developing and validating instruments to measure the effects of DHS deployment on HCSQ in real-life settings. By adapting and combining validated instruments, we provide a robust framework for assessing HCSQ, extending their application to the new and less-explored context of digital health.

Currently, decision makers face significant challenges in comparing outcomes from digital health implementation research because of the lack of consensus on evaluation tools, the use of many different instruments tailored to specific digital use cases, and the variance in the quality of the applied instruments. Although existing tools for HCSQ have not yet been used to assess the effects of DHSs, instruments for digital tools often evaluate relevant HCSQ factors but lack broad applicability.

We propose a comprehensive set of dimensions to measure HCSQ based on the validated instruments HEALTHQUAL, OPEQ, and PSQ-18. This set can be used by decision makers and practitioners to assess ongoing or completed DHS deployments. However, future research is essential to validate the reliability and validity of this scale further and to collect a robust database that can support pre-deployment decision-making in the future.

In summary, DigiHEALTHQUAL aims to fill the gap in current digital health evaluation practices, providing a standardized approach that enhances the comparability of research outcomes and ultimately supports better decision-making in the implementation of DHSs.