Original Paper
Abstract
Background: The integration of digital health solutions (DHSs) into health care systems has the potential to significantly enhance service delivery and health outcomes. Despite their benefits, the adoption remains slow, especially in outpatient care, and is hindered by various barriers, such as unclear effectiveness and high costs.
Objective: This study aimed to address the uncertainties regarding the cost-benefit ratio of DHSs by developing a comprehensive instrument to evaluate their impact on health care service quality across diverse settings (eg, across different diseases or types of DHSs).
Methods: We conducted a multistaged rapid review and semistructured, qualitative interviews to identify and adapt existing instruments evaluating the effects of DHSs. The first rapid review screened 4957 records and included 40 relevant papers to identify instruments currently used for DHS assessment after their deployment, yielding a total of 126 reported outcomes. Subsequently, we conducted interviews with 19 health care practitioners across 4 countries to validate and refine the 7 health care service quality dimensions derived from merging the Outpatient Experience Questionnaire (OPEQ), selected after the first rapid review, and Health Care Service Quality (HEALTHQUAL), an established instrument for measuring health care service quality derived from gray literature. On the basis of the results of the interviews, a second rapid review with 35 papers out of 493 screened records was conducted to identify instruments used to measure patient satisfaction, yielding a total of 29 patient satisfaction instruments.
Results: From the first rapid review, OPEQ was selected out of 18 relevant instruments identified among the 126 reported outcomes and combined with HEALTHQUAL. The interviews with health care professionals confirmed the relevance of all 7 health care service quality dimensions derived from OPEQ and HEALTHQUAL. In addition, 4 interviewees mentioned patient satisfaction as a further dimension missing in the framework presented during the interviews. From the subsequent rapid review, the Patient Satisfaction Questionnaire-Short Form was selected out of 6 relevant instruments identified among the 29 identified patient satisfaction instruments. By combining HEALTHQUAL, OPEQ, and Patient Satisfaction Questionnaire-Short Form, we derived the Digital Healthcare Service Quality (DigiHEALTHQUAL) questionnaire, which consists of 51 items across 8 dimensions, including accessibility, efficiency, empathy, general satisfaction, degree of improvements of care services, information, safety, and tangibles.
Conclusions: The DigiHEALTHQUAL questionnaire aims to provide a standardized approach for assessing the impact of DHSs on health care service quality across various use cases, therapeutic areas, and perspectives, facilitating comparison between DHSs and supporting decision makers in resource allocation and implementation decisions. Future research will focus on validating the DigiHEALTHQUAL in real-life settings and further refining it to comprehensively encompass both patient and health care practitioner perspectives.
doi:10.2196/68276
Keywords
Introduction
Theoretical Background
The integration of digital health solutions (DHSs) into health care has become an area of extensive research [,]. For our study, we adopted the European Commission’s definition of DHSs, which describes them as “tools and services that use information and communication technologies to improve prevention, diagnosis, treatment, monitoring and management of health-related issues and to monitor and manage lifestyle-habits that impact health” []. The many ways in which DHSs can transform health care have been shown in various studies, such as achieving cost and time savings [], improving medical safety and treatment adherence [], enhancing service quality [], and ameliorating health outcomes [].
Despite these benefits, the adoption of DHSs in health care is slow []. This is even more pronounced in outpatient settings compared to inpatient settings [,]. The key obstacles include required workflow adjustments, high costs for installation or maintenance, and unclear or limited effectiveness [,].
These challenges, especially the considerations between effectiveness and costs, underscore the importance of reliably measuring the impact of DHSs. Reliable measurement can support the further adoption of DHSs, that is, by helping decision makers compare the effects of different DHSs, select the most suitable DHS for their organization, and monitor the progress of deployments [-].
Moreover, as the landscape of DHSs grows increasingly complex with more and more digital use cases, tools, and providers [], it is becoming more difficult for decision makers to navigate this landscape. Therefore, robust evaluation frameworks become more important for their support []. Recent literature reveals that the instruments currently used to assess the implementation of DHSs often lack robust psychometric properties, leading to a lack of consensus on preferred assessment methodologies [-]. Existing validated instruments tend to focus on specific use cases or diseases, which limits their utility for comparison across multiple DHSs and health care settings [-]. This variation in quality and scope of reported effects complicates the task for decision makers attempting to compare, select, and implement DHSs.
Objectives
To address the limitations of currently applied instruments, it is essential to develop and establish comprehensive instruments. Existing studies have used a range of broadly applicable instruments, for example, measuring costs, resource use, patient satisfaction, and quality of life []. For example, the Mobile Healthcare App Database uses the Mobile Application Rating Scale to increase transparency regarding the quality of content and data security of >1300 DHSs [,].
Although service quality is a well-established and extensively studied outcome in various sectors, including health care—with numerous validated instruments available []—to our knowledge, established instruments for health care service quality (HCSQ) evaluation have not yet been used to assess the effects of DHSs. Recognizing the necessity for a versatile outcome measure, we considered whether HCSQ might be an effective metric to assess the effects of DHSs in a variety of health care settings. Therefore, this study aimed to address the lack of broadly applicable and reliable instruments by identifying, adapting, or developing a questionnaire that can measure the service quality of DHSs across different settings leveraging existing instruments. The broad adoption of such a tool would be particularly valuable for decision makers when considering the selection and implementation of DHSs.
Methods
Overview
This study used a sequential approach (
Figure 1) following predefined construction criteria to create a comprehensive questionnaire that can measure HCSQ in settings with different DHSs, across therapeutic areas, and from both patient and health care practitioner (HCP) perspectives for the assessment of the effects of DHS deployment on HCSQ.
We selected Health Care Service Quality (HEALTHQUAL) []—a validated, health care–specific service quality instrument based on the well-established service quality model Service Quality (SERVQUAL) []—for further adaptation to the context of this study. As HEALTHQUAL has not previously been used in digital health settings, we conducted a rapid review adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [], as well as recommendations from Tricco et al [] and King et al []. This review helped identify existing instruments used to assess the postdeployment effects of DHSs. The Outpatient Experience Questionnaire (OPEQ) was selected for the subsequent steps [,].
Neither the original HEALTHQUAL nor OPEQ have been validated for use with HCPs. To address this limitation, we conducted semistructured qualitative interviews with 19 HCPs, following the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist []. This process validated the relevance of the included dimensions for HCPs and led to the addition of a dimension for patient satisfaction, which was derived from the Patient Satisfaction Questionnaire-Short Form (PSQ-18) after an additional literature review on patient satisfaction measurement instruments [].
Consequently, we derived our suggested set of dimensions and items, hereafter referred to as Digital Healthcare Service Quality (DigiHEALTHQUAL).
Construction Criteria for the Questionnaire
To identify, adapt, or develop a questionnaire, we defined the following five criteria (Figure 1):
- Enables quantitative assessment of the postdeployment effects of DHSs following the definition of Proctor et al [] for implementation outcomes
- Is broadly applicable and suitable across therapeutic areas
- Incorporates multiple perspectives, that is, those of patients and HCPs, including health professionals, health associate professionals, and health management and support personnel, as defined by the World Health Organization (WHO) []
- Is applicable in outpatient practice settings, following the definition provided by the National Cancer Institute []
- Can be used for different digital use cases, that is, those already prioritized for the next research stage following this work (online appointment booking platforms, digital anamnesis tools, video consultations, and electronic patient records)
Selection of Research Framework
One well-established tool in the area of health service is SERVQUAL [], which is widely used to measure service quality across different sectors []. SERVQUAL has often been adapted to measure HCSQ in various health care settings [,]. This led to the development of several health care–specific variations of SERVQUAL. The 3 frequently used variations are called HEALTHQUAL and were developed by Miranda et al [], Lee [], and Mosadeghrad and Sokhanvar []. Despite evidence that DHSs impact HCSQ [,], none of these HEALTHQUAL instruments have been used to assess these effects. Therefore, this study aimed to fill this gap by adapting HEALTHQUAL by Lee [] to evaluate the impact of DHSs on HCSQ.
For this study, we used HEALTHQUAL by Lee [] because of its comprehensive design approach, which incorporated input from experts and patients as well as international health care service accreditation systems. HEALTHQUAL by Lee [] closely aligns with the original SERVQUAL framework and adds a dimension for health outcomes, making it a robust tool for assessing HCSQ []. In addition, HEALTHQUAL by Lee [] has been widely applied internationally in both inpatient and outpatient settings (eg, South Korea, India, Syria, and Ghana) [-]. The instrument consists of 5 dimensions with a total of 32 items. These dimensions are empathy, tangibles, safety, efficiency, and the degree of improvements of care services.
Multistaged Rapid Review
Overview
We conducted 2 rapid literature reviews following the PRISMA-RR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Rapid Reviews) guidelines and recommendations for rapid reviews outlined by Tricco et al [] and King et al []. As PRISMA-RR is still under development, we adapted PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines (especially PRISMA-ScR [Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews]) to align with the scope of this project. The search strategy and selection criteria are detailed below. More details can be found in and . Owing to the larger number of abstracts retrieved after title screening in the rapid review stage 1, we used ASReview LAB (Zenodo), an artificial intelligence–assisted screening tool, to expedite the abstract screening process []. This step was terminated upon meeting predefined stopping criteria in line with recent literature [,], specifically after screening at least 50% (422/843) of the abstracts and encountering at least 50 consecutive irrelevant abstracts. Data from the included studies were synthesized by extracting details from the reported findings and the instruments applied. This information was used to select suitable instruments for the following instrument development.
Rapid Review Stage 1
Overview
We conducted our search across 3 databases: PubMed, Scopus, and APA PsycINFO. Our search used a structured string comprising 4 conceptual blocks: digital health, implementation, effects, and assessment. The search terms were adapted from recent reviews within this field [,,,,-]. Articles were selected according to the criteria described in the following sections.
Inclusion and Exclusion Criterion 1: Type of Article
The articles were included if they reported the outcomes of a comprehensive data collection and analysis process. The articles were excluded if they (1) focused on describing the study design or methodology, (2) were statements, (3) reported single case studies, or (4) were conference papers.
Inclusion and Exclusion Criterion 2: Reported Outcomes
The articles were included if they (1) reported quantitative measures to assess postdeployment effects of DHSs and (2) included outcomes beyond health-related outcomes. The articles were excluded if they (1) focused on qualitative methods, (2) focused on intervention development and implementation, (3) focused on health-related outcomes, or (4) focused on describing the current status and trends of the health care sector in this area.
Inclusion and Exclusion Criterion 3: Type of Use Cases
Articles were included if they described the postdeployment outcome of a DHS that (1) belonged to 1 of the 4 selected digital use cases, (2) was implemented in an outpatient setting, and (3) did not require specific hardware. Articles were excluded if (1) the articles did not describe a DHS within the defined scope, (2) the DHS was deployed in an inpatient setting, or (3) the DHS required specific hardware.
Rapid Review Stage 2
Overview
We limited our search to the PubMed database after observing in rapid review stage 1 that it yielded the most relevant results for our research. Our search used a structured string comprising 3 conceptual blocks: patient satisfaction, assessment, and primary health care. The articles were selected according to the criteria described in the following sections.
Inclusion and Exclusion Criterion 1: Type of Article
The same criteria applied in rapid review stage 1 were used here to determine eligible article types.
Inclusion and Exclusion Criterion 2: Reported Outcomes
The articles were included if they contained patient satisfaction measures as at least 1 of the reported findings. The articles were excluded if they did not include a patient satisfaction measure.
Inclusion and Exclusion Criterion 3: Reported Health Care Setting
The articles were included if the study was conducted in a primary health care setting. The articles were excluded if the study was conducted in a different setting.
Inclusion and Exclusion Criterion 4: Reported Instruments
The articles were included if patient satisfaction was assessed using a validated instrument or if they referenced another publication as the source for the instrument. The articles were excluded if the applied instrument was (1) self-constructed, (2) not validated, or (3) did not reference previous scientific work.
HCP Interviews
We conducted qualitative, semistructured interviews with 19 HCPs. These interviews followed the COREQ checklist []. More details can be found in .
We designed an interview guide to validate findings from rapid review stage 1 and to explore additional HCSQ dimensions relevant to HCPs. More details are provided in . The interviews focused on two domains: (1) implementation and current experience with DHSs and (2) evaluation and further exploration of HCSQ dimensions.
Interviewees were recruited through targeted outreach to both contacts within the researchers’ network as well as to HCPs with relevant profiles and publicly available contact information. The following profiles were included: (1) working as an HCP [], (2) working in outpatient care [], and (3) considering the implementation or already using at least 1 of the selected DHS use cases. We ensured a diverse sample across specialties, roles, demographics, and countries.
The interviews were conducted with each interviewee individually over the phone. Each session was audio recorded and transcribed verbatim. Recruitment concluded after 19 interviews, upon reaching data saturation. The average duration of each interview was 29 (SD 8) minutes.
We performed coding and qualitative analysis using MAXQDA software (Verbi software) []. Our content analysis approach combined deductive and inductive methods []:
- Deductive coding established codes based on our rapid review stage 1 results to facilitate comparison.
- Inductive theme derivation derived additional themes inductively from the interview content if mentioned by >1 interviewee.
On the basis of these analyses, we quantitatively scored the relevance of each literature-derived dimension.
Ethical Considerations
The study protocol was reviewed and approved by the Ethics Committee of Witten/Herdecke University (S-113/2023). All interviewees received written information outlining the study’s objectives, procedures, and data handling practices and provided written informed consent before participation. Participation was entirely voluntary, and no incentives were offered. Audio recordings were deleted following verbatim transcription. During transcription and subsequent analysis, all personally identifiable information was removed. The interview data were factually anonymized, and results were presented only in aggregated form, making individual identification possible only with disproportionate effort. All data were stored on secure, password-protected systems.
Results
Rapid Review Stage 1
Our objective was to identify which instruments are currently used to measure the effects of DHSs. Our query yielded 9043 publications. To focus on the most current publications, we limited our inclusion to articles in English or German and published within the last 5 years, up to 2023. After removing duplicates, 4957 articles remained for screening, which resulted in 98 potentially relevant articles. Of these, 62 articles were accessible and underwent full-text screening, ultimately leading to the inclusion of 40 publications (
Figure 2).
In the 40 eligible studies, we identified a total of 126 reported outcomes. We applied the eligibility criteria used during literature screening to these outcomes analogously, resulting in 85 quantitative, nonhealth-related outcomes applied in outpatient settings measuring the postdeployment effects of selected digital use cases. Many of these were based on self-constructed instruments without reporting validity or reliability characteristics. Only 23 were based on instruments that have either been validated or have been used in previous publications [,-].
In total, we identified 18 distinct instruments (
Table 1). We assessed the instruments to ensure adherence to our predefined construction criteria, especially for applicability across digital use cases and perspectives. None of the instruments fulfilled both criteria; instead, 3 instruments assessed both the patient’s and HCPs’ perspectives but were use case–specific [,,,], and 9 instruments were not use case–specific but did not assess different perspectives [,-,,,,].
| Instrument | Applied to multiple perspectives | Not specific for a DHS use case | Mentions, n | Identified publications |
| ACESc | No | No | 1 | [] |
| CAHPSd,e | No | Yes | 2 | [,] |
| Deloitte US health care consumer survey | No | Yes | 1 | [] |
| GMCf Patient Questionnaire | No | Yes | 1 | [] |
| HCCQg | No | Yes | 1 | [] |
| JSPPPEh | No | Yes | 1 | [] |
| MISS-21i | No | No | 1 | [] |
| OPEQe,j | No | Yes | 2 | [,] |
| PEPPIk | No | Yes | 1 | [] |
| STOHFLAl | No | Yes | 1 | [] |
| Smartphone Use Score | Yes | No | 1 | [] |
| SUSm | Yes | No | 1 | [] |
| TAMn | No | Yes | 1 | [] |
| TESSo | No | No | 1 | [] |
| TUQp | Yes | No | 3 | [,,] |
| TMPQq | No | No | 1 | [] |
| TSUQr | No | No | 1 | [] |
| Trust in Physician Scale | No | No | 1 | [] |
aDHS: digital health solution.
bA total of 18 instruments were identified and reviewed for adherence to the predefined criteria; compliance with the following criteria was ensured through the previous selection steps: (1) postdeployment evaluation, (2) non–health related, and (3) application in an outpatient setting; open criteria were checked: applicability across multiple perspectives and digital use cases; and the number of mentions is an additional criterion for further selection.
cACES: Ambulatory Care Experience Survey.
dCAHPS: Consumer Assessment of Health Care Providers and Systems.
eInstruments that were selected for further analysis.
fGMC: General Medical Council.
gHCCQ: Health Care Climate Questionnaire.
hJSPPPE: Jefferson Scale of Patient Perception of Physician Empathy.
iMISS-21: Medical Interview Satisfaction Scale.
jOPEQ: Outpatient Experience Questionnaire.
kPEPPI: Perceived Efficacy in Patient-Physician Interactions Scale.
lSTOHFLA: Short test of functional health literacy in adults.
mSUS: System Usability Score.
nTAM: Technology Acceptance Model.
oTESS: Telehealth Satisfaction Scale.
pTUQ: Telemedicine Satisfaction and Usefulness Questionnaire.
qTMPQ: Telemedicine Perception Questionnaire Score.
rTSUQ: Telehealth Usability Questionnaire.
For further instrument development, we hypothesized that adjusting an existing instrument from the patient perspective to the HCP perspective would require fewer changes to the original instrument than making a use case–specific instrument more generalizable.
To select an instrument, we focused on those that had been applied multiple times, which is the case for the Consumer Assessment of Health Care Providers and Systems (CAHPS) [,] and OPEQ [,]. We mapped their dimensions against the HCSQ dimensions outlined by Lee []. The comparison revealed that both CAHPS and OPEQ cover dimensions similar to those of HEALTHQUAL (
Figure 3 []). Notably, OPEQ covers a slightly broader range of HCSQ dimensions compared to HEALTHQUAL. Therefore, we selected OPEQ for further use, merged mapping dimensions with HEALTHQUAL, and added accessibility (merging clinic access and previsit communication) and information as additional HCSQ dimensions.
HCP Interviews
Given that HEALTHQUAL and OPEQ have not been previously used with HCPs, we conducted interviews with HCPs with two objectives: (1) to confirm existing and derive new dimensions for the evaluation of HCSQ from the perspective of HCPs and (2) to understand the importance of each dimension for the overall HCSQ.
We interviewed 19 HCPs with diverse backgrounds (
Figure 4). We derived a total of 11 HCSQ dimensions from the interviews, with a broad consensus for four of them (mentioned by at least 12/19, 60% of interviewees):
- Accessibility (17/19, 89%); HCPs agreed that being accessible via phone, email, and at the front desk is crucial.
- Efficient administration (16/19, 16%); HCPs highlighted the importance of efficient administrative processes to keep patient waiting times low.
- Availability of appointments (13/19, 68%); HCPs pointed out the significance of having appointments available within an acceptable timeframe.
- Patient interaction (12/19, 63%); HCPs emphasized the importance of spending enough time with patients and maintaining a close physician-patient relationship.
Other less frequently mentioned dimensions included providing information, the atmosphere in the practice, available medical offerings, and improved health conditions. These dimensions match those in HEALTHQUAL and OPEQ. Notably, 4 (21%) out of 19 interviewees explicitly mentioned patient satisfaction as a separate dimension of HCSQ not represented by the dimensions of HEALTHQUAL and OPEQ. The 2 dimensions not represented by the dimensions of HEALTHQUAL and OPEQ were mentioned but with low frequency. Two interviewees mentioned integrated care and the continuous effort of the practice to improve the service.
Furthermore, interviewees rated the relevance of the HEALTHQUAL and OPEQ dimensions (
Figure 5). We found that there was a consensus regarding the most relevant HCSQ dimension (safety) but more variation regarding the remaining dimensions. However, all dimensions received a rating of at least 7 on a scale of 10, so we considered all of them as relevant.
These findings confirmed the relevance of the selected HCSQ dimensions for HCPs. However, patient satisfaction was mentioned by >20% (4/19) of interviewees as an additional relevant dimension. Consequently, we conducted another rapid review to identify instruments measuring patient satisfaction in the literature to be added to our questionnaire.
Rapid Review Stage 2
The objective was to identify the instruments currently used to measure patient satisfaction. We decided to narrow down the scope of this review further to primary health care settings because the 4 mentions of patient satisfaction were made by HCPs working in primary health care. Our initial search resulted in 1573 publications. To focus on recent research, we included only articles written in English or German that were published within the last 5 years, up to 2023, reducing the dataset to 493 articles. Further screening narrowed the selection to 136 potentially relevant articles. Of these, 105 were accessible and subjected to full-text review, leading to the selection of 35 publications (
Figure 6).
In the 35 eligible studies, 29 different instruments related to patient satisfaction were used or mentioned (Tables 2 and 3). To select the most suitable instrument to add to our questionnaire, we evaluated the following criteria: (1) the frequency of use, (2) the internal consistency (ie, Cronbach α), and (3) the feasibility for integration (ie, number of items).
| Instrument (cluster) | Clustered instruments (if applicable) | Applied, n | Mentioned, n | References |
| PSQb,c | PSQ II, PSQ III, and PSQ-18d | 6 | 2 | [-] |
| CAHPSc,e | Clinician and group version, emergency department version, hospital version, and patient-centered medical home version | 6 | 0 | [-] |
| PCATc,f | —g | 3 | 1 | [-] |
| EUROPEPc,h | — | 2 | 2 | [,-] |
| GPAQc,i | GPAQ and GPAQ-R2j | 2 | 1 | [,,] |
| AAFPk-MGMAl Patient Satisfaction Surveyc | — | 2 | 0 | [,] |
| CQm Index | — | 1 | 0 | [] |
| Client Satisfaction Questionnaire | — | 1 | 0 | [] |
| Tool to Improve Quality of Health Care | — | 1 | 0 | [] |
| EORTCn Satisfaction with Care Questionnaire | — | 1 | 0 | [] |
| General Practitioner Patient Survey | — | 1 | 0 | [] |
| Medical Interview Satisfaction Scale | — | 1 | 0 | [] |
| Primary Care Consultation Questionnaire | — | 1 | 0 | [] |
| Paediatric Otolaryngology Telemedicine Satisfaction Survey | — | 1 | 0 | [] |
| PROMISo | — | 1 | 0 | [] |
| Patient Satisfaction Survey Questionnaire | — | 1 | 0 | [] |
| Patient Survey for Quality of Care Scale | — | 1 | 0 | [] |
| Satisfaction with Stroke Care Questionnaire | — | 1 | 0 | [] |
| Survey of Healthcare Experiences of Patients | — | 1 | 0 | [] |
| Telemedicine Satisfaction Questionnaire | — | 1 | 0 | [] |
| WHO SARAp | — | 1 | 0 | [] |
| Components of Primary Care Index | — | 0 | 2 | [,] |
| GMCq Patient Questionnaire | — | 0 | 1 | [] |
| General Practice Assessment Survey | — | 0 | 1 | [] |
| General Satisfaction Questionnaire | — | 0 | 1 | [] |
| SERVQUALr | — | 0 | 1 | [] |
| TESSs | — | 0 | 1 | [] |
| TSUQt | — | 0 | 1 | [] |
| TUQu | — | 0 | 1 | [] |
aA total of 29 instruments were identified in 35 papers; the identified instruments were reviewed in 2 waves; in wave 1, the selection was based on the frequency of use.
bPSQ: Patient Satisfaction Questionnaire.
cInstruments that were selected for further evaluation.
dPSQ-18: Patient Satisfaction Questionnaire-Short Form.
eCAHPS: Consumer Assessment of Health Care Providers and Systems.
fPCAT: Primary Care Assessment Tool.
gNot applicable.
hEUROPEP: European Task Force on Patient Evaluations of General Practice Care.
iGPAQ: General Practice Assessment Questionnaire.
jGPAQ-R2: General Practice Assessment Questionnaire Revised Version 2.
kAAFP: American Academy of Family Physicians.
lMGMA: Medical Group Management Association.
mCQ Index: Consumer Quality Index.
nEORTC: European Organization for Research and Treatment of Cancer.
oPROMIS: Patient-Reported Outcomes Measurement Information System.
pWHO SARA: World Health Organization Service Availability and Readiness Assessment.
qGMC: General Medical Council.
rSERVQUAL: Service Quality.
sTESS: Telehealth Satisfaction Scale.
tTSUQ: Telemedicine Satisfaction and Usefulness Questionnaire.
uTUQ: Telehealth Usability Questionnaire.
| Instrument (cluster) | Internal consistency (Cronbach α) | Items, n | Additional references |
| PSQb | 0.91c | ≥18 | [,] |
| CAHPSd | 0.61-0.91e | 32 | [] |
| PCATf | 0.59-0.94 | ≥81 | [,] |
| EUROPEPg | 0.81-0.95 | 23 | [,] |
| GPAQh | 0.86-0.97i | 19 | [] |
| AAFPj-MGMAk Patient Satisfaction Survey | >0.7 | 23 | [] |
aA total of 29 instruments were identified in 35 papers; the identified instruments were reviewed in 2 waves; in wave 2, selection was based on internal consistency and feasibility for integration.
bPSQ: Patient Satisfaction Questionnaire.
cPSQ-18. Patient Satisfaction Questionnaire-Short Form.
dCAHPS: Consumer Assessment of Health Care Providers and Systems.
eClinician group and patient-centered medical home version.
fPCAT: Primary Care Assessment Tool.
gEUROPEP: European Task Force on Patient Evaluations of General Practice Care.
hGPAQ: General Practice Assessment Questionnaire.
iGPAQ-R2: General Practice Assessment Questionnaire Revised Version 2.
jAAFP: American Academy of Family Physicians.
kMGMA: Medical Group Management Association.
A total of 6 instruments (accounting for different variations or versions of an instrument as 1) were used multiple times. All of them demonstrated overall acceptable Cronbach α values. However, CAHPS and Primary Care Assessment Tool were reported to have Cronbach α values <0.7. The Patient Satisfaction Questionnaires and the General Practice Assessment Questionnaire contained the shortest instruments, with 18 and 19 items, respectively. On the basis of these criteria, we selected the most recent version of the Patient Satisfaction Questionnaires, the PSQ-18, for inclusion in our questionnaire based on its widespread use in the literature, reliability, and feasibility.
DigiHEALTHQUAL Development
We developed DigiHEALTHQUAL by synthesizing 3 validated instruments—HEALTHQUAL, OPEQ, and PSQ-18—guided by insights from our rapid reviews and interviews in the previous steps. HEALTHQUAL measures HCSQ via 32 items across 5 dimensions (empathy, efficiency, safety, tangible, and degree of improvements of care services), with responses rated on a 5-point scale ranging from “very bad” to “very good” []. The OPEQ assesses outpatient experiences through 24 items in 6 dimensions (clinic access, communication, hospital standards, organization, previsit communication, and quality of information), using a 10-point scale with anchoring phrases for each end of the scale [,]. The PSQ-18 evaluates patient satisfaction via 18 items across 7 domains (accessibility and convenience, communication, financial aspects, general satisfaction, interpersonal manner, technical quality, and time spent with doctors) on a 5-point Likert-like scale ranging from “strongly agree” to “strongly disagree” [].
The 3 instruments were compared item by item to remove duplicates and ensure contextual applicability by adapting or removing items as needed (). Of the initially longlisted 74 items, 13 were adjusted to better fit the focus of our study, 24 were removed because of redundancy or to account for differences between national health care systems, and 1 new item was added because adjusting the existing question to fit the focus of this study was not feasible (ie, data security considerations).
Subsequently, the remaining dimensions of the 3 original instruments were merged based on content overlap where feasible. The resulting DigiHEALTHQUAL questionnaire consists of 51 items across 8 dimensions: accessibility, efficiency, empathy, general satisfaction, degree of improvements of care services, information, safety, and tangibles.
This current questionnaire draft is designed to use a response scale similar to that of the PSQ-18, a 6-point Likert scale ranging from “completely disagree” to “completely agree,” allowing calculation of a satisfaction score for each subdimension. This scale has proven effective in practice []. Consequently, all items were converted accordingly.
Discussion
Principal Findings
This study reveals key findings regarding the measurement of the effects of DHSs on HCSQ. First, there is a notable lack of consensus on standardized instruments to measure the effects, especially for nonhealth-related outcomes. We hypothesize that this lack of agreement is a reflection of the novelty and wide variety of available DHSs, which consequently makes it challenging for decision makers to compare the effects of different DHSs across diverse contexts.
In contrast, there is greater agreement on the instruments used to measure patient satisfaction, indicating that these tools are well established and widely accepted. This consensus provides a solid foundation for evaluating patient satisfaction in the context of DHSs, offering a reliable starting point for further research.
Finally, instruments typically used to measure HCSQ have not yet been used to assess the impact of DHSs. This gap underscores the need to adapt existing HCSQ instruments for digital use cases. Interestingly, some of the instruments identified in this study include dimensions relevant to HCSQ. This indicates that some level of assessment of HCSQ elements is already taking place, but it is not being explicitly reported as such.
Comparison to Prior Work
Previous research has highlighted a lack of consensus on instruments for measuring implementation research outcomes in health care, including DHS deployments, primarily because of insufficient psychometric properties [-]. Our study confirms this general observation and provides additional insights specific to DHS implementation research outcome measurement. We found that the lack of standardized instruments is particularly pronounced for nonhealth-related outcomes in DHS implementation studies. Many of these studies used self-constructed instruments that neither referenced previous work nor reported validity or reliability properties. This variability and lack of rigor make it challenging to compare findings across different studies and contexts. Consequently, our findings underscore the need to validate new or existing instruments for nonhealth-related outcomes to enhance the reliability and comparability of DHS implementation studies and generate a consensus within the research community related to the applicable instruments.
In our study, we identified instruments to measure the effects of DHSs and several instruments to measure HCSQ. However, none of the currently used instruments explicitly assess the intersection of these 2 areas: how HCSQ is influenced by DHS deployment. Recognizing this gap, we followed the recommendations of Boateng et al [] by choosing to adapt and combine existing instruments rather than create a questionnaire de novo. This approach also aligns with the experimental practice of modifying established instruments such as SERVQUAL to fit specific health care contexts, although it has not yet been widely applied to DHSs []. By following this adaptation strategy, we ensure the relevance and applicability of our instruments while maintaining the validity, reliability, and comparability of the results by leveraging well-established and previously validated instruments.
Our HCP interviews revealed the recommendation to add patient satisfaction as an additional dimension to the set of HCSQ dimensions derived from HEALTHQUAL and OPEQ. This recommendation is anecdotally already supported by our rapid review stage 1, for example, where the CAHPS instrument included a question on overall satisfaction. The literature consistently described that there is a correlation between HCSQ and patient satisfaction or happiness in health care settings [-]. Various studies identify factors influencing patient satisfaction. For example, Duc Thanh et al [] identified key factors affecting patient satisfaction, including satisfaction with facilities, service provision, information transparency and administrative procedures, accessibility, and staff interaction and communication. Similarly, Ferreira et al [] highlighted factors such as waiting times, information provided, cleanliness, communication with patients, and doctors’ characteristics. These factors affecting patient satisfaction align closely with the HCSQ dimensions outlined by Lee [], underscoring the interconnectedness of the metrics patient satisfaction and HCSQ and their collective impact on health care evaluation—supporting the inclusion of patient satisfaction in DigiHEALTHQUAL.
Strengths and Limitations
By leveraging existing, validated instruments rather than creating new ones, we ensure comparability with previous studies. This approach not only strengthens the validity and reliability of DigiHEALTHQUAL but also facilitates meaningful comparisons and meta-analyses for future research. In addition, our study aimed to add the HCP perspective to patient-focused instruments, providing a holistic view of HCSQ. This comprehensive approach ensures that the nuances and needs of both key stakeholder groups are considered in future research, enhancing the overall applicability and relevance of DigiHEALTHQUAL. Moreover, our approach aligns with the best practices for developing and validating scales for health research, as described by Boateng et al [], who describe 9 steps across 3 phases for the development of rigorous scales. Following their work, we completed the first phase, “item development,” comprising the 2 steps “identification of domain and item generation” and “content validity” in this study [].
To enable broad application, we excluded dimensions from the original instruments, for example, items related to financial aspects, such as “appropriate costs” or “reasonable medical expenses,” because of varying reimbursement across national health care systems. This may result in a lack of insights into selected aspects of HCSQ. Furthermore, the interviews informing our questionnaire design were conducted exclusively with HCPs from high-income countries, which may limit the applicability of the resulting questionnaire to countries with different health care systems. However, the interviews only informed the decision to keep the HCSQ dimensions derived from the validated instruments HEALTHQUAL and OPEQ and to add patient satisfaction as an additional dimension—also derived from a validated instrument used in international studies. Therefore, we estimate that the geographic bias of the interviewees is negligible at this stage of our research. It is important to note that our resulting country-unspecific instrument may have limitations in practice, where national regulatory frameworks influence the adoption of DHSs and data use [,].
Generalization issues also arise, as some selected digital use cases may not impact all included HCSQ dimensions. For example, we theorize that online appointment booking platforms are rather unlikely to affect health outcomes. Furthermore, nonprioritized DHSs may require other evaluation criteria. For instance, we hypothesize that the effects of Digital Health Applications will materialize mainly in health outcomes rather than HCSQ and, therefore, necessitate specific, targeted assessment tools. This underscores the need for researchers to balance between using broadly applicable instruments that facilitate comparison across studies and specific instruments that provide detailed insights.
Within the broader context of health care, it is important to note the following: merely assessing the effects of DHSs is insufficient to accelerate their adoption. Comprehensive implementation strategies are necessary to facilitate widespread use.
Moreover, while scaling the digitalization of health care can improve health care delivery, it may also reinforce or exacerbate existing disparities. Health care access and quality are already unevenly distributed, and DHSs risk exacerbating these existing inequalities [].
On the one hand, digital technologies can help mitigate disparities, for example, by improving access in underserved areas or personalizing care. On the other hand, they can also create new barriers for populations with limited digital literacy, lower socioeconomic status, or restricted access to technology. Studies show that the use of digital health tools correlates with sociodemographic factors such as age; income; education; and, critically, health and digital literacy [-].
This introduces 2 challenges. First, in the context of validating DigiHEALTHQUAL, it is important that diverse user groups are appropriately represented. Digitally literate individuals are easier to recruit through online channels, whereas individuals with limited digital access or skills may be systematically underrepresented, leading to bias in the findings. Second, beyond the scope of our work, there is a need for comprehensive planning that ensures equity of access. DHSs must be designed and implemented in a way that safeguards against creating additional barriers and addresses common user concerns such as data privacy, usability, and accessibility.
Practical Implication and Further Research
The implications for further research based on our study are threefold.
Scale Development and Evaluation
Following the best practices for developing and validating scales as outlined by Boateng et al [], it is necessary to complete the next phases of scale development and evaluation to ensure the validity and reliability of the DigiHEALTHQUAL questionnaire. This involves conducting validation studies with patients to test the resulting questionnaire and subsequently performing validation studies with HCPs to adjust DigiHEALTHQUAL to reflect their perspectives as well. The authors have already initiated this process by conducting preliminary face validity testing with 10 representatives from each target group—patients and HCPs. On the basis of their feedback, minor adjustments were made to improve clarity and contextual relevance. Full-scale quantitative validation studies are currently in preparation, aiming to collect statistically adequate samples for each group. These studies are expected to conclude by the end of 2025 and will include psychometric analyses to support refinement, including item reduction and dimensional validation. To ensure methodological rigor, the planned validation will apply exploratory and confirmatory factor analysis to assess the dimensional structure, internal consistency testing (eg, Cronbach α), and construct validity evaluation. Each quantitative study will aim to include a minimum of 200 participants per target group (patients and HCPs).
Data Collection From DHS Deployment Studies
Collecting data from DHS deployment studies will enable decision makers to retrospectively assess whether the impact they aimed for has been realized after deployment. This data collection is crucial for understanding the real-world effects of DHSs and refining the evaluation tools based on practical insights.
Meta-Analysis of Postdeployment Studies
Conducting meta-analyses across postdeployment studies using DigiHEALTHQUAL will allow decision makers to prospectively consider the effects of DHSs and make informed decisions before deployment. This is especially important in resource-constrained environments, such as those commonly encountered in health care, where efficient allocation of resources is critical.
To optimize the impact of this research, we suggest initially focusing on use cases with the highest current implementation rates. Our prioritization of 4 use cases can serve as an initial starting point. In addition, there should be a strong emphasis on the outpatient setting, as digital transformation significantly lags behind in outpatient care compared to inpatient care [,]. A robust evaluation tool would be particularly valuable in outpatient settings, where clear and actionable guidance is critical for choosing among the many available DHSs that cater to patients with diverse health needs. In these settings, decision makers are often physicians who must juggle multiple roles. We hypothesize that these physicians have limited capacity to make business decisions or compare DHS offerings, given the workforce shortages and growing demand for health care services. Therefore, the decision makers need a robust evaluation framework even more than larger health care organizations with dedicated functions in place.
Conclusions
Our study underscores the importance of developing and validating instruments to measure the effects of DHS deployment on HCSQ in real-life settings. By adapting and combining validated instruments, we provide a robust framework for assessing HCSQ, extending their application to the new and less-explored context of digital health.
Currently, decision makers face significant challenges in comparing outcomes from digital health implementation research because of the lack of consensus on evaluation tools, the use of many different instruments tailored to specific digital use cases, and the variance in the quality of the applied instruments. Although existing tools for HCSQ have not yet been used to assess the effects of DHSs, instruments for digital tools often evaluate relevant HCSQ factors but lack broad applicability.
We propose a comprehensive set of dimensions to measure HCSQ based on the validated instruments HEALTHQUAL, OPEQ, and PSQ-18. This set can be used by decision makers and practitioners to assess ongoing or completed DHS deployments. However, future research is essential to validate the reliability and validity of this scale further and to collect a robust database that can support pre-deployment decision-making in the future.
In summary, DigiHEALTHQUAL aims to fill the gap in current digital health evaluation practices, providing a standardized approach that enhances the comparability of research outcomes and ultimately supports better decision-making in the implementation of DHSs.
Data Availability
The datasets generated or analyzed during this study are available from the corresponding author on reasonable request.
Authors' Contributions
DR, LF, and SM developed the overarching research question and study design. DR performed the literature reviews; conducted, transcribed, and qualitatively analyzed the interviews; and developed the DigiHEALTHQUAL questionnaire. DR drafted the manuscript. SM, LF, and AM provided critical review. SM coordinated the project. All authors reviewed the final manuscript.
Conflicts of Interest
None declared.
Multimedia Appendix 3
COREQ checklist—health care practitioner interviews.
PDF File (Adobe PDF File), 206 KBMultimedia Appendix 5
Development of the Digital Healthcare Service Quality Questionnaire.
PDF File (Adobe PDF File), 211 KBReferences
- Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. Jan 18, 2011;8(1):e1000387. [FREE Full text] [CrossRef] [Medline]
- Eden R, Burton-Jones A, Scott I, Staib A, Sullivan C. Effects of eHealth on hospital practice: synthesis of the current literature. Aust Health Rev. Sep 2018;42(5):568-578. [CrossRef] [Medline]
- eHealth: digital health and care. Directorate-General for Health and Food Safety. URL: https://health.ec.europa.eu/ehealth-digital-health-and-care/overview_en [accessed 2024-05-24]
- Eze ND, Mateus C, Cravo Oliveira Hashiguchi T. Telemedicine in the OECD: an umbrella review of clinical and cost-effectiveness, patient experience and implementation. PLoS One. 2020;15(8):e0237585. [FREE Full text] [CrossRef] [Medline]
- Keasberry J, Scott IA, Sullivan C, Staib A, Ashby R. Going digital: a narrative overview of the clinical and organisational impacts of eHealth technologies in hospital practice. Aust Health Rev. Dec 2017;41(6):646-664. [CrossRef] [Medline]
- Brönneke JB, Debatin JF. [Digitalization of healthcare and its effects on quality of care]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. Mar 2022;65(3):342-347. [FREE Full text] [CrossRef] [Medline]
- Brands MR, Gouw SC, Beestrum M, Cronin RM, Fijnvandraat K, Badawy SM. Patient-centered digital health records and their effects on health outcomes: systematic review. J Med Internet Res. Dec 22, 2022;24(12):e43086. [FREE Full text] [CrossRef] [Medline]
- Iyanna S, Kaur P, Ractham P, Talwar S, Najmul Islam A. Digital transformation of healthcare sector. What is impeding adoption and continued usage of technology-driven innovations by end-users? J Bus Res. Dec 2022;153:150-161. [CrossRef]
- Rahurkar S, Vest JR, Finnell JT, Dixon BE. Trends in user-initiated health information exchange in the inpatient, outpatient, and emergency settings. J Am Med Inform Assoc. Mar 01, 2021;28(3):622-627. [FREE Full text] [CrossRef] [Medline]
- Neunaber T, Meister S. Digital maturity and its measurement of general practitioners: a scoping review. Int J Environ Res Public Health. Feb 28, 2023;20(5):e309. [FREE Full text] [CrossRef] [Medline]
- Scott Kruse C, Karem P, Shifflett K, Vegi L, Ravi K, Brooks M. Evaluating barriers to adopting telemedicine worldwide: a systematic review. J Telemed Telecare. Jan 2018;24(1):4-12. [FREE Full text] [CrossRef] [Medline]
- Weik L, Fehring L, Mortsiefer A, Meister S. Understanding inherent influencing factors to digital health adoption in general practices through a mixed-methods analysis. NPJ Digit Med. Feb 27, 2024;7(1):47. [FREE Full text] [CrossRef] [Medline]
- Brantnell A, Temiz S, Baraldi E, Woodford J, von Essen L. Barriers to and facilitators of the implementation of digital mental health interventions as perceived by primary care decision makers: content analysis of structured open-ended survey data. JMIR Hum Factors. Jun 26, 2023;10:e44688. [FREE Full text] [CrossRef] [Medline]
- Ross J, Stevenson F, Dack C, Pal K, May C, Michie S, et al. Developing an implementation strategy for a digital health intervention: an example in routine healthcare. BMC Health Serv Res. Oct 19, 2018;18(1):794. [FREE Full text] [CrossRef] [Medline]
- Essén A, Stern AD, Haase CB, Car J, Greaves F, Paparova D, et al. Health app policy: international comparison of nine countries' approaches. NPJ Digit Med. Mar 18, 2022;5(1):31. [FREE Full text] [CrossRef] [Medline]
- Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, Cohen AB. Digital health: a path to validation. NPJ Digit Med. 2019;2:38. [FREE Full text] [CrossRef] [Medline]
- Khadjesari Z, Boufkhed S, Vitoratou S, Schatte L, Ziemann A, Daskalopoulou C, et al. Implementation outcome instruments for use in physical healthcare settings: a systematic review. Implement Sci. Aug 18, 2020;15(1):66. [FREE Full text] [CrossRef] [Medline]
- Godinho MA, Ansari S, Guo GN, Liaw ST. Toolkits for implementing and evaluating digital health: a systematic review of rigor and reporting. J Am Med Inform Assoc. Jun 12, 2021;28(6):1298-1307. [FREE Full text] [CrossRef] [Medline]
- Clinton-McHarg T, Yoong SL, Tzelepis F, Regan T, Fielding A, Skelton E, et al. Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review. Implement Sci. Nov 08, 2016;11(1):148. [FREE Full text] [CrossRef] [Medline]
- Enam A, Torres-Bonilla J, Eriksson H. Evidence-based evaluation of eHealth interventions: systematic literature review. J Med Internet Res. Nov 23, 2018;20(11):e10971. [FREE Full text] [CrossRef] [Medline]
- Sakamoto JL, Carandang RR, Kharel M, Shibanuma A, Yarotskaya E, Basargina M, et al. Effects of mHealth on the psychosocial health of pregnant women and mothers: a systematic review. BMJ Open. Feb 15, 2022;12(2):e056807. [FREE Full text] [CrossRef] [Medline]
- Wang X, Hunter DJ, Vesentini G, Pozzobon D, Ferreira ML. Technology-assisted rehabilitation following total knee or hip replacement for people with osteoarthritis: a systematic review and meta-analysis. BMC Musculoskelet Disord. Nov 03, 2019;20(1):506. [FREE Full text] [CrossRef] [Medline]
- Messner EM, Terhorst Y, Barke A, Baumeister H, Stoyanov S, Hides L, et al. The German version of the Mobile App Rating Scale (MARS-G): development and validation study. JMIR Mhealth Uhealth. Mar 27, 2020;8(3):e14479. [FREE Full text] [CrossRef] [Medline]
- Stach M, Kraft R, Probst T, Messner EM, Terhorst Y, Baumeister H, et al. Mobile health app database - a repository for quality ratings of mHealth apps. In: Proceedings of the 33rd International Symposium on Computer-Based Medical Systems. 2020. Presented at: CBMS '20; July 28-30, 2020:427-432; Rochester, MN. URL: https://ieeexplore.ieee.org/document/9183243 [CrossRef]
- Endeshaw B. Healthcare service quality-measurement models: a review. J Health Res. Apr 07, 2020;35(2):106-117. [CrossRef]
- Lee D. HEALTHQUAL: a multi-item scale for assessing healthcare service quality. Serv Bus. Jun 13, 2016;11(3):491-516. [CrossRef]
- Parasuraman A, Zeithaml V, Berry L. SERVQUAL: a multiple- item scale for measuring consumer perceptions of service quality. J Retail. 2024:1988. [FREE Full text]
- Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. Mar 29, 2021;372:n160. [FREE Full text] [CrossRef] [Medline]
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. Oct 02, 2018;169(7):467-473. [FREE Full text] [CrossRef] [Medline]
- King VJ, Stevens A, Nussbaumer-Streit B, Kamel C, Garritty C. Paper 2: performing rapid reviews. Syst Rev. Jul 30, 2022;11(1):151. [FREE Full text] [CrossRef] [Medline]
- Garratt AM, Bjaertnes ØA, Krogstad U, Gulbrandsen P. The OutPatient Experiences Questionnaire (OPEQ): data quality, reliability, and validity in patients attending 52 Norwegian hospitals. Qual Saf Health Care. Dec 2005;14(6):433-437. [FREE Full text] [CrossRef] [Medline]
- Augestad KM, Sneve AM, Lindsetmo RO. Telemedicine in postoperative follow-up of STOMa PAtients: a randomized clinical trial (the STOMPA trial). Br J Surg. Apr 2020;107(5):509-518. [CrossRef] [Medline]
- Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. Dec 2007;19(6):349-357. [CrossRef] [Medline]
- Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). RAND Corporation. 1994. URL: https://www.rand.org/content/dam/rand/pubs/papers/2006/P7865.pdf [accessed 2025-05-29]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. Mar 2011;38(2):65-76. [FREE Full text] [CrossRef] [Medline]
- Classifying health workers: mapping occupations to the international standard classification. World Health Organization. 2019. URL: https://cdn.who.int/media/docs/default-source/health-workforce/dek/classifying-health-workers.pdf [accessed 2024-05-24]
- NCI dictionary of cancer terms. National Cancer Institute. URL: https://www.cancer.gov/publications/dictionaries/cancer-terms/def/outpatient [accessed 2024-05-24]
- Souca L. SERVQUAL- thirty years of research on service quality with implications for customer satisfaction. In: Proceedings of the 4th Edition of the International Conference on Marketing - From Information to Decision. 2011. Presented at: MID '11; October 28-29, 2011:420-429; Cluj-Napoca, Romania. URL: https://www.researchgate.net/publication/346470601_SERVQUAL_-Thirty_years_of_research_on_service_quality_with_implications_for_customer_satisfaction
- Darzi MA, Islam SB, Khursheed SO, Bhat SA. Service quality in the healthcare sector: a systematic review and meta-analysis. LBS J Manag Res. Jan 16, 2023;21(1):13-29. [CrossRef]
- Miranda FJ, Chamorro A, Murillo LR, Vega J. Assessing primary healthcare services quality in Spain: managers vs. patients perceptions. Serv Ind J. Nov 2010;30(13):2137-2149. [CrossRef]
- Mosadeghrad A, Sokhanvar M. Measuring quality of services in Tehran teaching hospitals using HEALTHQUAL instrument. Razi J Med Sci. 2018;25:1. [FREE Full text]
- Lee D, Kim KK. Assessing healthcare service quality: a comparative study of patient treatment types. Int J Qual Innov. Jan 25, 2017;3(1):93. [CrossRef]
- Barrios-Ipenza F, Calvo-Mora A, Velicia-Martín F, Criado-García F, Leal-Millán A. Patient satisfaction in the Peruvian Health Services: validation and application of the HEALTHQUAL scale. Int J Environ Res Public Health. Jul 15, 2020;17(14):943. [FREE Full text] [CrossRef] [Medline]
- Asiamah N, Frimpong Opuni F, Aggrey M, Adu-Gyamfi K. ADAPTED SERVQUAL: a health service quality scale incorporating indicators of sanitation and hygiene. Qual Manag Health Care. 2021;30(3):184-193. [CrossRef] [Medline]
- Allahham L, Mouselli S, Jakovljevic M. The quality of Syrian healthcare services during COVID-19: a HEALTHQUAL approach. Front Public Health. 2022;10:970922. [FREE Full text] [CrossRef] [Medline]
- Chauhan VA, Limbad SJ. Service quality assessment of government and private hospitals using extended HEALTHQUAL dimensions in Surat region. Int J Curr Res. 2023;11(8):889-898. [FREE Full text]
- Shaikh MI, Sarkar A. A comparison of patients’ satisfaction treated at public and private healthcare institutions under the health scheme using the HEALTHQUAL model. J Integr Med Public Health. Jan 17, 2024;2(2):75-81. [CrossRef]
- ASReview LAB - a tool for AI-assisted systematic reviews. zenodo. URL: https://zenodo.org/records/15466337 [accessed 2025-05-29]
- Campos DG, Fütterer T, Gfrörer T, Lavelle-Hill R, Murayama K, König L, et al. Screening smarter, not harder: a comparative analysis of machine learning screening algorithms and heuristic stopping criteria for systematic reviews in educational research. Educ Psychol Rev. Feb 08, 2024;36(1):e369. [CrossRef]
- König L, Zitzmann S, Fütterer T, Campos D, Scherer R, Hecht M. An evaluation of the performance of stopping rules in AI-aided screening for psychological meta-analytical research. Res Synth Methods. Nov 2024;15(6):1120-1146. [CrossRef] [Medline]
- Abdolkhani R, Petersen S, Walter R, Zhao L, Butler-Henderson K, Livesay K. The impact of digital health transformation driven by COVID-19 on nursing practice: systematic literature review. JMIR Nurs. Aug 30, 2022;5(1):e40348. [CrossRef] [Medline]
- Allen P, Pilar M, Walsh-Bailey C, Hooley C, Mazzucca S, Lewis CC, et al. Quantitative measures of health policy implementation determinants and outcomes: a systematic review. Implement Sci. Jun 19, 2020;15(1):47. [FREE Full text] [CrossRef] [Medline]
- Chatterjee A, Prinz A, Gerdes M, Martinez S. Digital interventions on healthy lifestyle management: systematic review. J Med Internet Res. Nov 17, 2021;23(11):e26931. [FREE Full text] [CrossRef] [Medline]
- Granja C, Janssen W, Johansen MA. Factors determining the success and failure of eHealth interventions: systematic review of the literature. J Med Internet Res. May 01, 2018;20(5):e10235. [FREE Full text] [CrossRef] [Medline]
- Krick T, Huter K, Domhoff D, Schmidt A, Rothgang H, Wolf-Ostermann K. Digital technology and nursing care: a scoping review on acceptance, effectiveness and efficiency studies of informal and formal care technologies. BMC Health Serv Res. Jun 20, 2019;19(1):400. [FREE Full text] [CrossRef] [Medline]
- Li E, Clarke J, Neves AL, Ashrafian H, Darzi A. Electronic health records, interoperability and patient safety in health systems of high-income countries: a systematic review protocol. BMJ Open. Jul 14, 2021;11(7):e044941. [FREE Full text] [CrossRef] [Medline]
- Odendaal WA, Anstey Watkins J, Leon N, Goudge J, Griffiths F, Tomlinson M, et al. Health workers' perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database Syst Rev. Mar 26, 2020;3(3):CD011942. [FREE Full text] [CrossRef] [Medline]
- Peters GM, Kooij L, Lenferink A, van Harten WH, Doggen CJ. The effect of telehealth on hospital services use: systematic review and meta-analysis. J Med Internet Res. Sep 01, 2021;23(9):e25195. [FREE Full text] [CrossRef] [Medline]
- Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci. Oct 26, 2016;11(1):146. [FREE Full text] [CrossRef] [Medline]
- Rouleau G, Gagnon MP, Côté J, Payne-Gagnon J, Hudson E, Dubois C. Impact of information and communication technologies on nursing care: results of an overview of systematic reviews. J Med Internet Res. Apr 25, 2017;19(4):e122. [FREE Full text] [CrossRef] [Medline]
- MAXQDA 2020. VERBI Software. Berlin.; 2020. URL: https://www.maxqda.com/blogpost/how-to-cite-maxqda
- Kuckartz U, Rädiker S. Qualitative Inhaltsanalyse. Methoden, Praxis, Computerunterstützung (Grundlagentexte Methoden). Weinheim, Germany. Beltz Juventa; 2022.
- Bennell KL, Lawford BJ, Metcalf B, Mackenzie D, Russell T, van den Berg M, et al. Physiotherapists and patients report positive experiences overall with telehealth during the COVID-19 pandemic: a mixed-methods study. J Physiother. Jul 2021;67(3):201-209. [FREE Full text] [CrossRef] [Medline]
- Bruno B, Mercer MB, Hizlan S, Peskin J, Ford PJ, Farrell RM, et al. Virtual prenatal visits associated with high measures of patient experience and satisfaction among average-risk patients: a prospective cohort study. BMC Pregnancy Childbirth. Apr 06, 2023;23(1):234. [CrossRef] [Medline]
- Buvik A, Bugge E, Knutsen G, Småbrekke A, Wilsgaard T. Patient reported outcomes with remote orthopaedic consultations by telemedicine: a randomised controlled trial. J Telemed Telecare. Sep 2019;25(8):451-459. [CrossRef] [Medline]
- Danila MI, Sun D, Jackson LE, Cutter G, Jackson EA, Ford EW, et al. Satisfaction with modes of telemedicine delivery during COVID-19: a randomized, single-blind, parallel group, noninferiority trial. Am J Med Sci. Nov 2022;364(5):538-546. [FREE Full text] [CrossRef] [Medline]
- Darr A, Senior A, Argyriou K, Limbrick J, Nie H, Kantczak A, et al. The impact of the coronavirus (COVID-19) pandemic on elective paediatric otolaryngology outpatient services - An analysis of virtual outpatient clinics in a tertiary referral centre using the modified paediatric otolaryngology telemedicine satisfaction survey (POTSS). Int J Pediatr Otorhinolaryngol. Nov 2020;138:110383. [FREE Full text] [CrossRef] [Medline]
- Horsham C, Snoswell C, Vagenas D, Loescher LJ, Gillespie N, Soyer HP, et al. Is teledermoscopy ready to replace face-to-face examinations for the early detection of skin cancer? Consumer views, technology acceptance, and satisfaction with care. Dermatology. 2020;236(2):90-96. [CrossRef] [Medline]
- Kim SM, Kim T, Cha WC, Lee JH, Kwon IH, Choi Y, et al. User experience of mobile personal health records for the emergency department: mixed methods study. JMIR Mhealth Uhealth. Dec 15, 2020;8(12):e24326. [FREE Full text] [CrossRef] [Medline]
- Orrange S, Patel A, Mack WJ, Cassetta J. Patient satisfaction and trust in telemedicine during the COVID-19 pandemic: retrospective observational study. JMIR Hum Factors. Apr 22, 2021;8(2):e28589. [CrossRef] [Medline]
- Rickwood D, Wallace A, Kennedy V, O'Sullivan S, Telford N, Leicester S. Young people's satisfaction with the online mental health service eheadspace: development and implementation of a service satisfaction measure. JMIR Ment Health. Apr 17, 2019;6(4):e12169. [FREE Full text] [CrossRef] [Medline]
- Tan NG, Yang LW, Tan MZ, Chng J, Tan MH, Tan C. Virtual care to increase military medical centre capacity in the primary health care setting: a prospective self-controlled pilot study of symptoms collection and telemedicine. J Telemed Telecare. Sep 2022;28(8):603-612. [CrossRef] [Medline]
- Thomas E, Lee CM, Norman R, Wells L, Shaw T, Nesbitt J, et al. Patient use, experience, and satisfaction with telehealth in an Australian population (reimagining health care): web-based survey study. J Med Internet Res. Aug 17, 2023;25:e45016. [FREE Full text] [CrossRef] [Medline]
- Walker J, Leveille S, Bell S, Chimowitz H, Dong Z, Elmore JG, et al. OpenNotes after 7 years: patient experiences with ongoing access to their clinicians' outpatient visit notes. J Med Internet Res. May 06, 2019;21(5):e13876. [FREE Full text] [CrossRef] [Medline]
- Watson L, Qi S, Delure A, Link C, Photitai E, Chmielewski L, et al. Virtual cancer care during the COVID-19 pandemic in Alberta: evidence from a mixed methods evaluation and key learnings. JCO Oncol Pract. Sep 2021;17(9):e1354-e1361. [FREE Full text] [CrossRef] [Medline]
- Zammit M, Siau R, Williams C, Hussein A. Patient satisfaction from ENT telephone consultations during the coronavirus disease 2019 pandemic. J Laryngol Otol (Forthcoming). Nov 17, 2020:1-6. [FREE Full text] [CrossRef] [Medline]
- Alsayali MM, Al-Sahafi A, Mandoura N, Usman Shah HB, Abdul Rashid OA, AlSharif K, et al. Patients' satisfaction after primary health care centers' integration with Ministry of Health Hospitals, Jeddah. J Epidemiol Glob Health. Jun 2019;9(2):135-142. [FREE Full text] [CrossRef] [Medline]
- Hayek S, Derhy S, Smith ML, Towne SD, Zelber-Sagi S. Patient satisfaction with primary care physician performance in a multicultural population. Isr J Health Policy Res. Mar 25, 2020;9(1):13. [FREE Full text] [CrossRef] [Medline]
- Leow HT, Liew SM. A cross sectional study on patient satisfaction and its association with length of consultation at the University Malaya Medical Centre Primary Care Clinic. Malays Fam Physician. Jul 24, 2022;17(2):71-80. [FREE Full text] [CrossRef] [Medline]
- Li H, Swe KM, Jabbar MA, Ching SM. Patients' satisfaction on primary healthcare services in China and its associated factors during COVID-19 pandemic: a cross-sectional study. Malays Fam Physician. 2023;18:4. [FREE Full text] [CrossRef] [Medline]
- Mauro E, Marciano S, Torres MC, Roca JD, Novillo AL, Gadano A. Telemedicine improves access to hepatology consultation with high patient satisfaction. J Clin Exp Hepatol. 2020;10(6):555-562. [FREE Full text] [CrossRef] [Medline]
- Staub MB, Pellegrino R, Gettler E, Johnson MC, Roumie CL, Grijalva CG, et al. Association of antibiotics with veteran visit satisfaction and antibiotic expectations for upper respiratory tract infections. Antimicrob Steward Healthc Epidemiol. 2022;2(1):e100. [FREE Full text] [CrossRef] [Medline]
- Abass G, Asery A, Al Badr A, AlMaghlouth A, AlOtaiby S, Heena H. Patient satisfaction with the emergency department services at an academic teaching hospital. J Family Med Prim Care. Apr 2021;10(4):1718-1725. [FREE Full text] [CrossRef] [Medline]
- Alkahtani AS, Abbas AH, Bin Rsheed AM, Alabood AF, Alqahtani AA. Assessing the impact of a service excellence program on improving patient experience at primary health care centers. Cureus. Aug 2023;15(8):e43749. [FREE Full text] [CrossRef] [Medline]
- Hung DY, Mujal G, Jin A, Liang SY. Patient experiences after implementing lean primary care redesigns. Health Serv Res. Jun 2021;56(3):363-370. [FREE Full text] [CrossRef] [Medline]
- Jordan JN, Wadsworth TG, Robinson R, Hruza H, Paul A, O'Connor SK. Patient satisfaction with pharmacist-provided health-related services in a primary care clinic. Pharmacy (Basel). Nov 21, 2021;9(4):187. [FREE Full text] [CrossRef] [Medline]
- Legler A, Price M, Parikh M, Nebeker JR, Ward MC, Wedemeyer L, et al. Effect on VA patient satisfaction of provider's use of an integrated viewer of multiple electronic health records. J Gen Intern Med. Jan 2019;34(1):132-136. [FREE Full text] [CrossRef] [Medline]
- Zakare-Fagbamila RT, Park C, Dickson W, Cheng TZ, Gottfried ON. The true penalty of the waiting room: the role of wait time in patient satisfaction in a busy spine practice. J Neurosurg Spine. Jul 01, 2020;33(1):95-105. [CrossRef] [Medline]
- Albahrani S, Albidy H, Alomar N, Bin Mutreb L, Alkhofi A, Alsaleh Z, et al. Patient satisfaction with primary healthcare services in Al-Ahsa, Saudi Arabia. Cureus. Nov 2022;14(11):e31478. [FREE Full text] [CrossRef] [Medline]
- Kijima T, Matsushita A, Akai K, Hamano T, Takahashi S, Fujiwara K, et al. Patient satisfaction and loyalty in Japanese primary care: a cross-sectional study. BMC Health Serv Res. Mar 25, 2021;21(1):274. [FREE Full text] [CrossRef] [Medline]
- Pitrou I, Berbiche D, Vasiliadis HM. Mental health and satisfaction with primary care services in older adults: a study from the patient perspective on four dimensions of care. Fam Pract. Sep 05, 2020;37(4):459-464. [CrossRef] [Medline]
- Wang W, Haggerty J. Development of primary care assessment tool-adult version in Tibet: implication for low- and middle-income countries. Prim Health Care Res Dev. Jul 01, 2019;20:e94. [FREE Full text] [CrossRef] [Medline]
- Chen W, Feng Y, Fang J, Wu J, Huang X, Wang X, et al. Effect of trust in primary care physicians on patient satisfaction: a cross-sectional study among patients with hypertension in rural China. BMC Fam Pract. Sep 21, 2020;21(1):196. [FREE Full text] [CrossRef] [Medline]
- Ferreira PL, Raposo V, Tavares AI. Primary health care patient satisfaction: explanatory factors and geographic characteristics. Int J Qual Health Care. Apr 27, 2020;32(2):93-98. [CrossRef] [Medline]
- Sparkes SP, Atun R, Bӓrnighausen T. The impact of the family medicine model on patient satisfaction in Turkey: panel analysis with province fixed effects. PLoS One. 2019;14(1):e0210563. [FREE Full text] [CrossRef] [Medline]
- Mohamoud G, Mash R. Evaluation of the quality of service delivery in private sector, primary care clinics in Kenya: a descriptive patient survey. S Afr Fam Pract (2004). Oct 22, 2020;62(1):e1-12. [FREE Full text] [CrossRef] [Medline]
- Alshowair A, Altamimi S, Alruhaimi F, Tolba A, Almeshari A, Almubrick R, et al. Assessment of primary health care specialized reference clinics in Riyadh first health cluster: outcome, cost-effectiveness and patient satisfaction. Clinicoecon Outcomes Res. 2022;14:371-381. [FREE Full text] [CrossRef] [Medline]
- Nowaskie DZ, Fogel RS, Fogel JM. Impact on patient satisfaction and importance of medical intake and office staff in a multidisciplinary, one-stop shop transgender program in Indianapolis, Indiana. J Multidiscip Healthc. 2019;12:665-673. [FREE Full text] [CrossRef] [Medline]
- Rutten MH, Giesen PH, Assendelft WJ, Westert G, Smits M. Effects of access to radiology in out-of-hours primary care on patient satisfaction and length of stay. Eur J Gen Pract. Dec 2021;27(1):221-227. [FREE Full text] [CrossRef] [Medline]
- Valero-Cantero I, Casals C, Espinar-Toledo M, Barón-López FJ, Martínez-Valero FJ, Vázquez-Sánchez MÁ. Cancer patients' satisfaction with in-home palliative care and its impact on disease symptoms. Healthcare (Basel). Apr 29, 2023;11(9):1272. [FREE Full text] [CrossRef] [Medline]
- Saric J, Kiefer S, Peshkatari A, Wyss K. Assessing the quality of care at primary health care level in two pilot regions of Albania. Front Public Health. 2021;9:747689. [FREE Full text] [CrossRef] [Medline]
- Perfors IA, Helsper CW, Noteboom EA, Visserman EA, van Dorst EB, van Dalen T, et al. Effects of structured involvement of the primary care team versus standard care after a cancer diagnosis on patient satisfaction and healthcare use: the GRIP randomised controlled trial. BMC Prim Care. Jun 04, 2022;23(1):145. [FREE Full text] [CrossRef] [Medline]
- Feng Y, Gravelle H. Patient self-reported health, clinical quality, and patient satisfaction in English primary care: practice-level longitudinal observational study. Value Health. Nov 2021;24(11):1660-1666. [FREE Full text] [CrossRef] [Medline]
- Eksteen LB, Mash RJ. Evaluating the validity and reliability of the Medical Interview Satisfaction Scale in South African primary care consultations. Fam Pract. May 23, 2019;36(3):310-316. [CrossRef] [Medline]
- Stark S, Worm L, Kluge M, Roos M, Burggraf L. The patient satisfaction in primary care consultation-Questionnaire (PiC): an instrument to assess the impact of patient-centred communication on patient satisfaction. PLoS One. 2021;16(7):e0254644. [FREE Full text] [CrossRef] [Medline]
- Lee CF, Wang XF, Wong PN, Koh YL, Ngoh SH, Mohtar ZM, et al. Psychometric evaluation of a patient satisfaction survey questionnaire to assess advanced practice nurse ambulatory services in primary care. J Nurs Manag. Oct 2020;28(7):1481-1488. [FREE Full text] [CrossRef] [Medline]
- Steyl T. Satisfaction with quality of healthcare at primary healthcare settings: perspectives of patients with type 2 diabetes mellitus. S Afr J Physiother. 2020;76(1):1321. [FREE Full text] [CrossRef] [Medline]
- Abdul Aziz AF, Tan CE, Ali MF, Aljunid SM. The adaptation and validation of the satisfaction with stroke care questionnaire (Homesat) (SASC10-My™) for use in public primary healthcare facilities caring for long- term stroke survivors residing at home in the community. Health Qual Life Outcomes. Jun 20, 2020;18(1):193. [FREE Full text] [CrossRef] [Medline]
- Satpathy S, Wundaville LT, Satapathy S, Malik A, Singh S, Singh AR, et al. A systematic review of patient satisfaction scales and their applicability to COVID-19 hospitalized patients: gaps and emerging needs. J Patient Exp. 2022;9:23743735221079132. [FREE Full text] [CrossRef] [Medline]
- Care coordination measures atlas update: chapter 6. Measure maps and profiles (continued, 3). Agency for Healthcare Research and Quality. URL: https://www.ahrq.gov/ncepcr/care/coordination/atlas/chapter6b.html [accessed 2025-05-29]
- Pasarín MI, Berra S, González A, Segura A, Tebé C, García-Altés A, et al. Evaluation of primary care: the "primary care assessment tools - facility version" for the Spanish health system. Gac Sanit. Jan 2013;27(1):12-18. [FREE Full text] [CrossRef] [Medline]
- Shi L, Starfield B, Xu J. Validating the adult primary care assessment tool. J Fam Pract. 2001;50(2):161. [FREE Full text]
- Grol R, Wensing M, Mainz J, Jung HP, Ferreira P, Hearnshaw H, et al. European Task Force on Patient Evaluations of General Practice Care (EUROPEP). Patients in Europe evaluate general practice care: an international comparison. Br J Gen Pract. Nov 2000;50(460):882-887. [FREE Full text] [Medline]
- Dimova R, Stoyanova R, Keskinova D. The EUROPEP questionnaire for patient's evaluation of general practice care: Bulgarian experience. Croat Med J. Feb 28, 2017;58(1):63-74. [FREE Full text] [CrossRef] [Medline]
- Mead N, Bower P, Roland M. The General Practice Assessment Questionnaire (GPAQ) - development and psychometric characteristics. BMC Fam Pract. Feb 20, 2008;9:13. [FREE Full text] [CrossRef] [Medline]
- White B. Measuring patient satisfaction: how to do it and why to bother. Fam Pract Manag. Jan 1999;6(1):40-44. [FREE Full text] [Medline]
- Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149. [FREE Full text] [CrossRef] [Medline]
- Fatima I, Humayun A, Iqbal U, Shafiq M. Dimensions of service quality in healthcare: a systematic review of literature. Int J Qual Health Care. Feb 01, 2019;31(1):11-29. [CrossRef] [Medline]
- Ferreira DC, Vieira I, Pedro MI, Caldas P, Varela M. Patient satisfaction with healthcare services and the techniques used for its assessment: a systematic literature review and a bibliometric analysis. Healthcare (Basel). Feb 21, 2023;11(5):378. [FREE Full text] [CrossRef] [Medline]
- Duc Thanh N, My Anh BT, Xiem CH, Quynh Anh P, Tien PH, Thi Phuong Thanh N, et al. Patient satisfaction with healthcare service quality and its associated factors at one polyclinic in Hanoi, Vietnam. Int J Public Health. 2022;67:1605055. [FREE Full text] [CrossRef] [Medline]
- Munawarah S, Arifin S, Febriana SK. Examining the associations between service quality, educational level, occupational background, and patient satisfaction in healthcare facilities. River Stud. Aug 10, 2023;1(2):104-118. [CrossRef]
- Al Meslamani AZ. Technical and regulatory challenges of digital health implementation in developing countries. J Med Econ. 2023;26(1):1057-1060. [FREE Full text] [CrossRef] [Medline]
- Lawrence K. Digital Health Equity. Brisbane, Australia. Exon Publication; 2022.
- Li S, Cui G, Yin Y, Xu H. Associations between health literacy, digital skill, and eHealth literacy among older Chinese adults: a cross-sectional study. Digit Health. 2023;9:20552076231178431. [FREE Full text] [CrossRef] [Medline]
- Nakayama LF, Binotti WW, Link Woite N, Fernandes CO, Alfonso PG, Celi LA, et al. The digital divide in Brazil and barriers to telehealth and equal digital health care: analysis of internet access using publicly available data. J Med Internet Res. Jul 21, 2023;25:e42483. [FREE Full text] [CrossRef] [Medline]
- Cornejo Müller A, Wachtler B, Lampert T. [Digital divide-social inequalities in the utilisation of digital healthcare]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. Feb 2020;63(2):185-191. [FREE Full text] [CrossRef] [Medline]
Abbreviations
| CAHPS: Consumer Assessment of Health Care Providers and Systems |
| COREQ: Consolidated Criteria for Reporting Qualitative Research |
| DHS: digital health solution |
| DigiHEALTHQUAL: Digital Healthcare Service Quality |
| HCP: health care practitioner |
| HCSQ: health care service quality |
| HEALTHQUAL: Health Care Service Quality |
| OPEQ: Outpatient Experience Questionnaire |
| PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses |
| PRISMA-RR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Rapid Reviews |
| PRISMA-ScR: Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews |
| PSQ-18: Patient Satisfaction Questionnaire-Short Form |
| SERVQUAL: Service Quality |
Edited by A Mavragani; submitted 01.11.24; peer-reviewed by O Allela; comments to author 06.02.25; revised version received 25.05.25; accepted 25.05.25; published 24.07.25.
Copyright©Dominik Rigo, Leonard Fehring, Achim Mortsiefer, Sven Meister. Originally published in JMIR Formative Research (https://formative.jmir.org), 24.07.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.