Prototype Tool for Assessing the Quality and Safety of Virtual Consultations in Primary Care: Qualitative Formative Evaluation

Introduction

Virtual consultations (VCs), including telephone and video consultations, are now an integral part of primary care delivery. By mid-2021, approximately 46% of patients worldwide had accessed health care remotely, an increase from just 11% before the COVID-19 pandemic [1]. This transformation prompted international organizations, such as the World Health Organization, to issue extensive guidance on digital health, with more than 60 documents developed between 2020 and 2024 [1], while nearly 40% of countries have formalized VCs within their national health strategies [2]. As VCs become embedded within health systems, concerns about how to ensure and evaluate their quality and safety have grown [1,3].

In the United Kingdom and many other high-income countries, general practitioners (GPs) remain the main health care professionals of VCs. Despite their central role in ensuring the quality and safety of virtual care, existing evaluation mechanisms largely focus on user satisfaction or technological functionality rather than clinical quality or patient safety [3,4]. Furthermore, the insights of frontline practitioners are rarely incorporated into the development of assessment tools, limiting their relevance and uptake in real-world practice [5].

Despite prior policy emphasis on digital health, there is limited understanding of how to assess the quality and safety of VCs in practice [1]. Foundational frameworks, such as the Institute of Medicine’s 6 domains of health care quality and the Donabedian structure-process-outcome model, provide conceptual guidance for assessing health care quality and performance, but they are not readily adapted to the dynamic, relational, and technological nature of VCs [6-9]. Existing studies and reviews continue to emphasize access, efficiency, or patient satisfaction, with little focus on clinical safety, diagnostic quality, or equity outcomes [10-12]. Moreover, implementation frameworks, such as the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework and the Consolidated Framework for Implementation Research (CFIR), offer system-level insights but lack operational guidance for day-to-day primary care workflows [13-16].

The Planning and Evaluating Remote Consultation Services framework began to address this gap by considering clinical relationships, staff, home environment, technologies, organizational context, and the dynamic interactions among them over time [17]. Furthermore, we previously developed a conceptual framework outlining 6 core domains for assessing the quality and safety of VCs in primary care. These domains include technology infrastructure, health care professional expertise, patient awareness, safety and risk management, policy support, and monitoring and evaluation. This framework was derived from an extensive literature review and a global survey of more than 1600 physicians across 20 countries [18]. In addition, a subsequent Delphi study validated these domains and indicators through international expert consensus [19]. Through 3 rounds of surveys, high levels of consensus were reached regarding which domains were key for successful VC implementation and which components required further attention. While this previous work established the proposed content and structure of the framework, its relevance, usability, and applicability for intended users had not yet been explored.

Therefore, this study aimed to explore GPs’ perspectives on the relevance and usability of a prototype tool for evaluating the quality and safety of VCs and to inform its further refinement and development for use across primary care systems to guide, monitor, and evaluate the implementation of VCs and ensure that they are safe and high quality. Specifically, it sought to evaluate GPs’ perceptions of (1) the tool’s relevance and how it could be used, (2) the preferred delivery format, and (3) suggestions to enhance the tool’s relevance and adoption in routine practice.

Methods

Study Design

This study used a qualitative, exploratory research design to conduct a formative evaluation of a prototype tool for assessing the quality and safety of VCs in primary care. We followed the COREQ (Consolidated Criteria for Reporting Qualitative Research) to support clear and comprehensive reporting of the study (Multimedia Appendix 1) [20].

Study Population

This study captured the views of GPs with experience delivering VCs in primary care settings. Participants were eligible if they were GPs, either fully qualified or in training; working in primary care settings; and had experience conducting VCs via telephone, video, or asynchronous online platforms. No minimum recency or frequency of VC experience was specified as an eligibility criterion. Recent VC activity was characterized by participants’ reported number and proportion of VCs conducted in the preceding 12 months. Participants were also required to have sufficient English fluency to participate in an online group discussion conducted in English.

Sampling and Recruitment

Participants were recruited using convenience and snowball sampling strategies; no purposive sampling was applied. A recruitment poster and invitation email were distributed via the researchers’ contact networks, professional networks, academic collaborators, and social media. The final sample comprised GPs practicing in the United Kingdom (n=7, 50%), Portugal (n=3, 21.4%), Türkiye (n=2, 14.3%), Greece (n=1, 7.1%), and Ireland (n=1, 7.1%). Recruitment took place between May and June 2025.

Prototype Assessment Tool

A conceptual framework for assessing the quality and safety of VCs in primary care was previously developed by our team. The framework was derived from a global survey of more than 1600 physicians across 20 countries [18] and validated through international expert consensus [19]. A mock web-based prototype was subsequently developed to operationalize this framework as a practice-level assessment tool. The prototype was not designed to deliver VCs; rather, it was intended to support structured assessment of how VCs were implemented within primary care organizations.

The interface organized assessment statements across 6 domains: contextual considerations, technology infrastructure, awareness and experience, safety and risk management, strategic planning and supporting policies, and monitoring and evaluation. Within each domain, users reviewed a series of statements relating to the implementation of virtual care and indicated whether each criterion was met using a binary yes or no response, with the option to provide a justification (Figure 1A). Example statements addressed whether patients and health care professionals were willing and able to use virtual care, whether contextual barriers to VC access were considered, and whether alternative face-to-face pathways were available when virtual care was not appropriate or preferred.

Responses were summarized at the domain level and visualized using a radar chart to provide an overview of areas in which recommendations had been achieved and areas requiring further attention (Figure 1B). The prototype also included functions for saving assessments and generating or exporting reports. Thus, unlike platforms used to conduct VCs, the prototype functioned as an assessment and quality improvement resource rather than as a system for delivering the consultation itself.

‎

Data Collection

A total of 3 online focus groups were conducted in June 2025 via Microsoft Teams, comprising groups of 5, 5, and 4 participants; group allocation was based primarily on participants’ availability. The final sample size of 14 reflected the number of eligible GPs who were recruited and available to participate during the recruitment period, with group size determined pragmatically by scheduling and feasibility considerations. The first focus group was facilitated by ALN, a clinical associate professor in digital health with extensive research experience in digital health. The subsequent 2 focus groups were facilitated by WZ, with support from SS. WZ had received formal postgraduate training in qualitative research methods, and SS had prior experience in conducting and analyzing qualitative health research. Sessions were held in English, lasted approximately 60 minutes, and were recorded and transcribed. A brief prediscussion survey was used to collect participants’ characteristics. The mock-up prototype of the tool (Figure 1) was presented during the session to prompt feedback on usability, format, and integration into clinical practice and served as a stimulus for feedback on the prototype and to inform its subsequent refinement.

A topic guide was used to facilitate all focus group discussions (Multimedia Appendix 2). The guide was divided into 3 main sections. The first section aimed to gather background information on participants’ professional roles and experience with VCs. The second section focused on obtaining feedback to inform refinement of the practical assessment tool, including participants’ views on optimal timing, usability, and preferred delivery format. The final section invited participants to share any additional reflections, suggestions for improvement, or relevant experiences that might inform future development of the tool.

All study materials were stored on password-protected servers at Imperial College London. Only members of the research team had access to the recordings and transcripts. No follow-up focus groups were conducted.

Data Analysis

Transcripts were analyzed using a combined deductive and inductive thematic approach in NVivo 15 (Lumivero) [21]. Meaningful segments of text relevant to the study aims were treated as the unit of coding. Initial deductive codes were informed by the topic guide and study objectives, while additional inductive codes were generated from concepts emerging directly from the data. Two researchers with backgrounds in public health and clinical medicine (SS and WZ) independently coded transcripts. Coding was subsequently compared, and differences in code interpretation or application were discussed and resolved through consensus. The coding framework was iteratively refined throughout the analysis, with overlapping codes merged and code definitions clarified where appropriate. Related codes were grouped into subthemes and subsequently developed into higher-order themes through discussion within the research team [22].

Ethical Considerations

Ethics approval for this study was granted by the Imperial College Research Ethics Committee (ICRER 7756479). All participants provided informed electronic consent before participation. Study materials were stored on password-protected servers at Imperial College London and were accessible only to members of the research team. Transcripts were anonymized before analysis to protect participant confidentiality. Participants received £30 (£1=US $1.34 as of October 21, 2026) in total for participation in the focus group, comprising £25 for 1 hour of participation and £5 for working-from-home or internet expenses.

Results

Participant Characteristics

A total of 14 GPs from 5 countries (the United Kingdom, Portugal, Türkiye, Greece, and Ireland) took part in 3 online focus groups, with an average duration of 46.6 (SD 10.3) minutes. Most participants were female (n=11, 78.6%), younger than 50 years (n=13, 92.9%), and identified as White (n=12, 85.7%; Table 1). Just over half of the participants (n=9, 64.3%) were fully qualified GPs, while the remainder were trainees. Half of the participants (n=7, 50%) had conducted more than 100 VCs in the past year. Telephone remained the dominant modality (n=13, 92.9%), followed by e-consultation forms (n=4, 28.6%), video (n=3, 21.4%), and messaging platforms (n=1, 7.1%). Most participants (n=12, 85.7%) were based in urban settings.

Perceived Relevance of the Tool

This theme captured participants’ views on the perceived value of the tool, its potential use for reflection and quality improvement, the preferred frequency of use, and opportunities for multidisciplinary engagement.

Positive assessments of the proposed evaluation tool were expressed across all 3 focus groups, particularly in relation to reflective practices and quality improvement. Two participants explicitly described the tool as “comprehensive” and “fundamental”:

I found this tool comprehensive and useful to check quality.
[Türkiye, FG3, GP1]

This is a fundamental tool for improving quality.
[Portugal, FG1, GP5]

Audit, reflection, or quality improvement were identified as potential use cases by more than half of participants. The tool was also considered potentially valuable for professional development, including portfolio-based reflection and appraisal. Some participants stated the following:

I think every GP trainee would love to have access to this tool and just do a quick quality improvement activity that they could then reflect on in their portfolio.
[United Kingdom, FG1, GP2]

I think maybe perhaps for an audit or maybe self-reflection tool, might be useful.
[Ireland, FG2, GP3]

What I see the best use of this would be as an audit tool. For somebody to do an audit, whether it’s going to be one of the partners or the practice manager or a few people involved.
[United Kingdom, FG2, GP2]

In total, 6 participants proposed specific intervals for repeat assessment, with suggestions ranging from monthly to annual use; 6-monthly or annual review was more commonly proposed. Participants linked less frequent assessment to the need to avoid administrative burden and allow sufficient time for changes to be implemented:

Maybe annually, or biannually at maximum, probably annually.
[United Kingdom, FG2, GP5]

Six months is better than three months because if they're doing audits too many times, it's going to be overwhelming for them as well.
[Türkiye, FG3, GP2]

Participants generally viewed the tool as more suitable for practice-level or shared use than for assessment by individual clinicians. Suggested users included practice managers, GP partners, multidisciplinary teams, and patient representatives, reflecting a preference for incorporating multiple perspectives rather than relying solely on an individual clinician’s assessment. One participant explained it as follows:

The best way to fill out this tool would be to have a multidisciplinary team (MDT): a GP partner or trainee, the manager, and then a patient advocate.
[United Kingdom, FG1, GP2]

Views on Usability, Content, and Structure

This theme focused on the adaptability of the tool across different health care contexts, preferred response formats, perceived workload, and factors that could support or hinder its use in routine practice.

Participants offered several suggestions on the structure and content of the tool. A consistent theme was the need for flexibility across different health care systems. Given variations in national structures, resources, and digital readiness, many participants felt a “one-size-fits-all” approach would be ineffective. Others stressed the importance of allowing contextual differentiation, particularly for use at the facility or system level. For example, some participants shared the following:

I believe it should be adapted to the culture of the country in which it’s going to be used.
[Greece, FG1, GP3]

I think this tool can be relatively easily used in a system like the UK. But in other systems or countries, they might struggle to use it at the facility level.
[United Kingdom, FG1, GP1]

Six participants considered the binary yes or no response format too restrictive and proposed more flexible response options, including Likert scales and open-ended responses. Some participants stated the following:

That yes or no is quite hard to answer, (...) it’s always a spectrum.
[United Kingdom, FG1, GP1]

Maybe a Likert style or any other option could be more flexible for the doctors.
[Türkiye, FG3, GP2]

Three participants also suggested incorporating routinely available quantitative indicators to support interpretation of the assessment, such as rates of repeated telephone consultations or conversion from virtual to face-to-face consultations. One participant stated the following:

Some metrics about repeated telephone appointments for specific patients, or how many telephone appointments were then converted into face-to-face appointments, would be brilliant- just to add to the interpretation of the results.
[United Kingdom, FG1, GP2]

Additional workload and time burden emerged as widespread concerns. Completing the tool would add to their existing workload, particularly if not clearly aligned with appraisal or quality improvement requirements. One GP described it as “hidden workload,” stating the following:

We don’t get paid extra to do this.
[United Kingdom, FG2, GP1]

Another participant stated the following:

I’d have to make sure that it’s really simple and it’s not adding any additional time to my day.
[Ireland, FG2, GP3]

Moreover, participants suggested that uptake could be improved through alignment with existing professional incentives, such as revalidation, audits, or financial support. One UK participant specifically noted that if the tool were not linked to the Quality and Outcomes Framework (QOF) or another remunerated activity, it was unlikely to be prioritized in routine practice. Some other participants stated the following:

This tool should be announced with some incentives…people should evaluate themselves and re-evaluate like twice a year.
[Türkiye, FG3, GP2]

You will have to either pay them to do it or make a very convincing case that, this is going to improve efficiency.
[Greece, FG3, GP3]

If it’s not part of QOF, or the practice doesn't get paid to do it, then it's not going to be a priority for the practice necessarily.
[United Kingdom, FG2, GP5]

Preferred Formats and Delivery Modes

This theme addressed participants’ preferred modes of delivery, shared access across practice teams, and integration with existing digital systems and clinical workflows.

Six participants preferred a web-based format as the primary mode of delivery. Key perceived advantages included desktop accessibility, shared practice access, and the ability to retain and compare assessments over time. Mobile apps were discussed as a complementary option, particularly for providing reminders. Some participants expressed as follows:

I would say an online site. With that, everyone in the practice could do the questionnaire.
[Portugal, FG1, GP5]

I think a website could be nice because this is like the mainstream medium for all of us. When we're doing our work, it’s from a computer most of the time.
[Türkiye, FG3, GP2]

I agree with the website. But I think that an application has the benefit that you can send notifications, so that's kind of a reminder.
[Portugal, FG3, GP4]

Integration with existing electronic health record (EHR) systems or clinical systems was also a recurring recommendation. Participants suggested that integration could reduce additional logins and duplicate data entry, facilitate access to routinely collected information, and make the assessment easier to incorporate into existing workflows:

If it were integrated into our electronic medical record systems, that would be great.
[United Kingdom, FG1, GP4]

If it’s integrated into the system of your electronic health record system, then you can easily see those important details of how to improve yourself, if it’s possible.
[Türkiye, FG3, GP2]

The ideal would be if it integrates with your clinical system so that you don’t have to log in again.
[United Kingdom, FG3, GP3]

Discussion

Summary of Key Findings

This study explored GPs’ perspectives as part of a formative evaluation of a prototype tool for guiding, monitoring, and evaluating the safe and high-quality implementation of VCs in primary care. Overall, participants regarded the prototype as potentially useful for reflection, audit, and quality improvement and generally envisaged it as a periodic, shared practice-level resource rather than a mechanism for continuous monitoring of individual clinicians. They also emphasized the need for flexible response formats (eg, Likert scales and open-text fields) and contextual adaptability, together with integration into existing EHR systems and alignment with appraisal, quality improvement, or incentive mechanisms to minimize additional burden.

Comparison With Existing Research

Previous research has addressed both the development and implementation of VC systems in primary care. Studies of online consultation systems have examined how these services are introduced into general practice, how they interact with existing clinical workflows, and how patients and staff experience their use [23,24]. Systematic reviews have further examined the use, acceptability, and experiences of electronic, telephone, and video consultations among patients and health care professionals [22,25]. These studies have been important in identifying the practical and organizational factors that shape the quality and safety of virtual care.

Building on this evidence, a range of frameworks, indicators, and evaluation approaches have been developed to organize relevant considerations into structured domains or measures. For example, the Planning and Evaluating Remote Consultation Services framework captures the clinical, organizational, technological, relational, and ethical complexity of remote consultations [17], while other work has focused on dimensions of health care quality, telehealth performance indicators, or patient safety measures [10,11,26]. Broader virtual care quality frameworks have also sought to organize relevant dimensions of quality into structured domains [27], while a recent scoping review identified a wide range of quality measures used in ambulatory virtual care and highlighted variation in the domains addressed across existing measures [28]. At an organizational and health system level, the World Health Organization Telehealth Quality of Care Tool provides a structured resource to support assessment, reflection, and quality improvement in telehealth services [29]. However, these approaches provide substantial guidance on what aspects of virtual care quality should be considered and how they may be organized for assessment but provide more limited evidence on how a primary care–specific assessment resource is perceived and used by GPs as intended end users. Rather than examining a system used to deliver VCs, this study evaluates a resource intended to assess the quality and safety of VC implementation. More specifically, it examines how intended GP users perceive the relevance, usability, and potential implementation of a concrete primary care assessment prototype and how such a resource might need to be structured and embedded for routine use.

The findings of this study are broadly consistent with wider digital health implementation research. Usability, workflow fit, interoperability, and compatibility with existing EHR systems have previously been identified as important influences on the adoption and sustainability of digital technologies [30-34]. Similarly, additional workload and limited incentives reflect recognized barriers to digital tool use in primary care [35-37]. The contribution of this study therefore lies less in identifying previously unrecognized implementation barriers than in showing how these established considerations translate into requirements for the design and implementation of a VC quality and safety assessment tool. Taken together, the study adds user-focused formative evidence at the prototype evaluation stage, complementing existing evidence on VC delivery systems, quality frameworks, and measurement approaches.

Limitations

A strength of this study was the direct involvement of GPs, as intended end users, in evaluating the relevance, usability, and potential implementation of the prototype. Importantly, the inclusion of participants from multiple countries offers an additional strength, providing early insight into system-level variation that may shape priorities and use, and supporting the potential transferability of the tool across settings.

However, this study has several limitations. The participant group was small (N=14) and skewed toward White, urban, and early-career clinicians, with limited input from rural or resource-constrained contexts. In addition, the use of convenience and snowball sampling may have limited the breadth of perspectives represented and should be considered when interpreting the transferability of the findings. Conducting the focus groups in English and using voluntary recruitment may have favored those with stronger digital health interests. Finally, the prototype was intentionally illustrative, limiting feedback on technical specifics but enabling open discussion of relevance, usability, and workflow integration.

Implications for Practice, Policy, and Future Research

The findings identify several practical priorities for the further development and implementation of the prototype. For developers, future iterations should support more flexible response formats, shared access across practice teams, and integration with existing EHR or clinical systems to minimize duplicate data entry and additional workload. The tool should also allow sufficient contextual adaptation to accommodate differences in health care organization, digital infrastructure, and local practice.

At the practice and organizational level, the findings support periodic, team-based use rather than continuous completion by individual clinicians. Embedding the tool within existing clinical governance meetings, audit, or quality improvement processes may promote shared responsibility while reducing the risk of creating an additional administrative task. At the policy level, alignment with existing appraisal, quality improvement, or incentive mechanisms may support uptake and sustained use. If subsequently validated for routine use, aggregated assessments could also contribute to broader monitoring of virtual care quality and safety and inform service- or system-level improvement.

Future research should focus on iterative development, piloting, and validation of a fully functional prototype across diverse settings. Testing must consider variations in EHR systems, digital infrastructure, interoperability, staffing models, and national policy contexts. Stakeholder involvement should extend beyond GPs to include nurses, managers, patients, policymakers, and IT teams. Longitudinal evaluation will be essential to assess impact on clinical practice, patient outcomes, and organizational learning, providing evidence to inform national digital health quality strategies and support safer, more equitable digital transformation.

Conclusions

This study explored GPs’ perspectives on a prototype tool for assessing VC quality and safety and identified priorities for its further refinement and implementation. Findings suggest support for a structured yet adaptable resource that can support reflective practice, quality improvement, and clinical appraisal. Participants emphasized flexible response formats, integration with existing systems, and alignment with appraisal and incentive structures, highlighting the importance of minimizing additional workload and embedding evaluation within routine quality improvement and audit processes. Importantly, the tool was viewed not as a top-down performance measure but as a resource to empower clinicians and support shared responsibility for VC quality and safety. By involving GPs directly, this study helps bridge the policy-level digital transformation agenda with frontline needs, providing a foundation for further development and testing and for embedding evaluation mechanisms in everyday digital health care delivery.