Diagnostic accuracy in primary care is critical, yet missed, delayed, or incorrect diagnoses remain common in ambulatory care and contribute to avoidable morbidity, fragmented follow-up, and increased health care costs [1,2]. Diagnostic difficulty is particularly relevant in primary care, where generalist physicians often evaluate broad, undifferentiated symptom constellations with limited testing at the point of care [3]. Conditions with subtle, nonspecific, or atypical presentations, such as pulmonary embolism or systemic autoimmune disease, are especially vulnerable to delayed or missed diagnosis [4,5]. These challenges create a rationale for tools that support hypothesis generation, information gathering, and diagnostic verification during the consultation.

Large language models (LLMs) have shown strong diagnostic performance in medical question-answering and curated vignette benchmarks, and recent systems increasingly move from single-answer generation toward conversational or workflow-level diagnostic support [6-8]. However, evidence that LLMs improve physician performance is mixed. In a randomized vignette-based study, GPT-4 access did not significantly improve physicians’ diagnostic reasoning compared with conventional resources, despite strong standalone model performance [9]. A later randomized trial found a more modest improvement in patient care tasks when GPT-4 support was embedded in a sequential information environment [10]. More recent evidence suggests that gains may depend on structured AI-literacy training or collaborative diagnostic workflows rather than model access alone [11,12].

Safety evaluation is equally important. LLMs can generate incorrect, unsupported, or biased outputs, and prior studies have documented hallucination, demographic bias, and cognitive bias sensitivity in medical LLM outputs [13-15]. Human-artificial intelligence (AI) interaction studies also show that incorrect AI recommendations can degrade clinician performance even when correct recommendations improve it [16,17]. Interface design is therefore central: Explanations, confidence cues, and recommendation framing may influence trust, but explainability alone does not reliably prevent overreliance [18-20].

Most prior diagnostic support studies have relied on static text vignettes or structured tasks rather than real-time, voice-based consultations. This distinction matters because diagnostic reasoning during a consultation evolves dynamically as physicians ask questions, test hypotheses, and decide what information is relevant. Voice-based assistants and ambient clinical tools are increasingly entering clinical workflows, but much of the existing evidence focuses on documentation, operational efficiency, or constrained clinical tasks rather than diagnostic accuracy and overreliance during an unfolding consultation [21-24]. LLM-powered virtual patients provide a controlled way to evaluate these interactions before exposure to real patients, preserving experimental control while approximating some features of clinical dialogue [25,26].

Against this background, we conducted a formative, case-level randomized, adjudicator-blinded simulation study to evaluate a real-time, voice-based diagnostic decision-support system during high-difficulty simulated primary care consultations. The study was designed as a diagnostic stress test rather than as an estimate of routine clinical effectiveness. By using interactive virtual-patient consultations, the study aimed to move beyond static vignette testing while preserving a controlled setting in which diagnostic benefit and overreliance risk could be evaluated before prospective clinical deployment.

The primary objective was to assess whether AI assistance to the physician was associated with higher Top-3 diagnostic accuracy than an unassisted simulation condition without external diagnostic aids. A secondary safety objective was to quantify whether physicians were more likely to submit diagnoses concordant with incorrect AI suggestions when those suggestions were visible. These objectives reflect 2 linked questions for formative evaluation: whether real-time AI support can improve diagnostic performance in a challenging simulated setting and whether visible incorrect suggestions create a measurable overreliance signal. Consultation duration, physician-reported usefulness, and overall satisfaction were assessed as formative workflow and acceptability measures; the remaining additional analyses were exploratory or sensitivity analyses.

We conducted a formative, case-level randomized, adjudicator-blinded simulation study in a web-based virtual primary care clinic. The study was designed as a high-difficulty diagnostic simulation stress test: Physicians evaluated diagnostically challenging virtual patient cases, and outcomes measured physician diagnostic performance, workflow, acceptability, and overreliance signals rather than patient health outcomes. We hypothesized that real-time, voice-integrated AI assistance would be associated with higher diagnostic accuracy than an unassisted simulation condition without external diagnostic aids, while also increasing the risk that physicians would submit diagnoses concordant with incorrect AI suggestions when those suggestions were visible.

Physicians conducted voice-based consultations with a patient simulator powered by the GPT-4o Realtime API for low-latency voice interaction. Through case-specific prompts, the model was anchored to predefined demographics, symptoms, and permitted primary care physical examination and point-of-care information. The simulator was instructed to behave as a patient rather than as a clinical assistant, to answer only physician-initiated questions, and not to disclose diagnostic labels or unsolicited clinical reasoning. To approximate a constrained primary care visit, it processed requests for predefined primary care point-of-care tools and physical examinations and declined advanced investigations such as imaging or laboratory workups. Pre-study testing was used to check adherence to this constrained role, and a retrospective assessment of that adherence was done based on the transcripts of the conversations. The architecture separated the patient simulator from the AI assistant, ensuring that the AI assistant processed only the live physician-patient dialogue and not the underlying vignette text nor simulator prompts (Figure 1).

For each incoming virtual-patient case, physicians were assigned either to real-time AI support or to the unassisted simulation condition. In the AI-assisted condition, diagnostic support was visible during the consultation; in the unassisted condition, physicians used the same virtual clinic but without AI support or external diagnostic aids. A panel of 3 independent adjudicators, all practicing family physicians, assessed the correctness of diagnoses. A diagnosis was considered correct based on predefined criteria that accepted exact matches, synonyms, or a broader clinical category only when the specific gold standard subtype could not be differentiated with the information available in a primary care setting. Final adjudication was determined by majority rule: When at least 2 of the 3 adjudicators provided the same judgment, their shared judgment was retained as the final classification; otherwise, the assessment was to be discarded.

Study registration was not applicable because this was a formative, randomized, simulation-based, physician performance study using virtual patient cases: No real patients were enrolled, no clinical care was delivered or modified, no identifiable patient health data were used, and no patient health outcomes or patient adverse events were measured. All outcomes were physician-level performance, time, usability, or automation-bias measures.

To support transparency and reproducibility, the full study protocol, reporting checklist, simulated clinical vignettes, dataset from all 260 virtual consultations, adjudicator evaluations, statistical analysis code and results, data dictionary, analytical transparency document, virtual patient performance audit, and sensitivity analysis outputs were made publicly available in a permanent repository [27]. The analytical transparency document maps the public statistical code and console outputs to the corresponding manuscript results, enabling end-to-end third-party verification from the locked adjudicator-derived dataset.

Although the study involved physician participants, it did not involve real patients, the delivery or modification of clinical care, identifiable patient health data, or patient health outcomes and therefore did not fall within the categories requiring ethics committee approval under applicable local regulations. Following completion of the study, the study documentation was reviewed by the Research Ethics Committee of Hospital Universitario La Paz (Madrid, Spain), which issued a formal institutional determination confirming that ethics approval was not required (Reference 57/230428.9/26, dated August 11, 2026). The processing of participating physicians’ personal data complied with Regulation (EU) 2016/679 and Spanish Organic Law 3/2018 [28,29]. Data related to participating physicians were de-identified and stored in an access-restricted Google Drive folder accessible only to the study authors; no directly identifying participant information was included in the adjudication files or the public repository.

All participating physicians provided written informed consent before taking part in the study. Recruitment and consent materials described the evaluated tool, the simulated case format, the expected time commitment, random assignment to AI-assisted and unassisted simulation conditions, and the assessment of diagnostic accuracy. Participation was voluntary, and participants could discontinue participation. Participants received no financial compensation. Access to the evaluated system during the simulation was provided at no cost to participants.

We selected 40 diagnostically challenging adult clinical vignettes from a peer-reviewed library that was originally validated through supermajority agreement by a panel of 7 independent primary care physicians [30]. This selection was intentional: The case set was used to stress-test diagnostic reasoning under conditions in which missed or delayed diagnoses are plausible, not to reproduce the prevalence or average difficulty of unselected primary care encounters. A physician and 2 LLM-based agents, which were built on a different architecture from the evaluated assistant, selected the cases to reduce content familiarity. The AI assistant was structurally blinded to the original vignette text and underlying simulator prompts, evaluating exclusively the de novo live dialogue dynamically steered by the physician.

Separately, physicians were recruited online through a URL shared with the Madrid regional health service (SERMAS) and social media. Eligibility required active clinical practice in Spain and specialty board certification; computer and internet literacy were implicit eligibility criteria because participation required use of a secure web application and voice interface. Recruitment materials described the evaluated tool, the simulated cases, the expected time commitment, and the AI-assisted and unassisted simulation conditions. A total of 13 board-certified family and community medicine physicians were accepted with no more than one per clinic to reduce cross-contamination. Each physician completed a 30-minute online orientation with the simulation interface before the study.

Computer-generated randomization occurred at the case level. For each physician, cases were randomly drawn from the 40-case pool and assigned with intended equal probability to either AI assistance or the unassisted simulation condition; no blocking or stratification by physician or case was used, so the achieved distribution could deviate from exact 1:1 balance by chance.

Physicians were necessarily unblinded to AI availability because diagnostic suggestions were visible in AI-assisted consultations and absent in unassisted consultations. The 3 independent adjudicators who assessed diagnostic correctness were blinded: They received only physicians’ diagnoses, AI assistant diagnostic suggestions (also generated in the background for the unassisted condition cases), and the vignette gold standard diagnoses, stripped of all metadata that could identify study arms or indicate whether the AI suggestion had been displayed to the physician. Data analysts worked from the adjudicators’ evaluations and were not blinded; however, they did not influence outcomes, as all statistical modeling was conducted on a locked dataset comprised strictly of the independent adjudicators’ blinded scorings. After blinded adjudication, analytical endpoints were generated through deterministic recoding of the locked adjudicator-derived labels; therefore, the analysts had no ability to alter the primary outcome grading. The statistical code, console outputs, and result-to-manuscript analytical transparency document are available in the public repository [27].

In both conditions, physicians used the same voice interface to conduct simulated consultations. The study was fully web-based and remote: Participants accessed a secure web application from home and were instructed to use a headset in an uninterrupted, quiet environment. To define a controlled, resource-restricted contrast, physicians were instructed not to use nonstudy external diagnostic aids during either condition. The unassisted comparator was intended to isolate the effect of real-time AI assistance under the study conditions and was not intended to represent resource-enabled usual care. After each consultation, physicians submitted between 1 and 3 final diagnoses, which were securely stored. Consultation duration, time stamps, arm assignment, case identifiers, and submitted diagnoses were captured by the web application. After completing their assigned cases, physicians completed an online questionnaire assessing perceived usefulness and overall satisfaction on 5-point response scales. All metadata and arm identifiers were stripped before blinded adjudication. The AI assistant was used under a license provided by the manufacturer.

In the AI-assisted condition, physicians received real-time decision support from the Medsys AI-Clinical Assistant V1.0.0 (Medsys AI, SL), the evaluated system, without any functionality change, content update, or bug fix release after study commencement. The system displayed diagnostic suggestions and information-gathering prompts during the consultation while leaving final diagnostic responsibility with the physician (Figure 2). It transcribed the physician-patient conversation and routed it to an orchestration subsystem, which queried a research agent for clinical guideline insights and a diagnostic agent for hypothetical diagnoses. The interface passively displayed a dynamic differential diagnosis with brief rationales, contextually relevant history-taking prompts, and physical examination suggestions. Physicians could use, ignore, or override any displayed content. The system integrates off-the-shelf speech-to-text and generative LLMs, specifically GPT-4o and GPT-4o-mini, which were called concurrently at high frequencies to maintain real-time, low-latency performance. No fine-tuning of these models was performed. Instead, the system relied on retrieval-augmented generation (RAG) anchored to a curated corpus of over 1000 clinical practice guidelines accepted by Spanish medical associations.

Because simulator behavior was central to the validity of the study, we conducted a retrospective transcript-based audit of virtual patient adherence to its role, assessing diagnostic leakage, excessive helpfulness, fluency or conversational blocking, and adherence to the required physical examination response format [27].

All hypothesis tests were 2-sided with α=.05. All randomized consultations were analyzed according to assigned condition. Case characteristics and case allocation by physician were summarized using counts and percentages. Physician sex was summarized using counts and percentages, whereas physician age and clinical experience were summarized using medians and IQRs. No imputation was required because all randomized consultations had complete outcome data.

The simulation was conducted between April 1, 2025, and May 20, 2025, during which 13 board-certified family and community medicine physicians from the Madrid regional health service (SERMAS) each evaluated 20 simulated patients through the web-based virtual clinic. Once all simulations were completed, the 3 independent adjudicators evaluated the correctness of the diagnoses.

Overall, 140 episodes were randomly assigned to the AI-assisted condition, and 120 episodes were randomly assigned to the unassisted simulation condition. This imbalance reflected chance variation from case-level randomization without blocking or stratification. All randomized consultations generated a submitted diagnosis; no consultation required repetition because of a reported technical failure. No episodes were lost or excluded after randomization, and all 260 randomized consultations were adjudicated and analyzed according to assigned condition (Figure 3). The baseline characteristics of the simulated patients are shown in Table 1.

Among the 13 participating physicians, 8 were women (62%) and 5 were men (38%); median age was 30.0 years (IQR 28.0‐43.0). Participants had a median of 6.0 (IQR 4.0‐19.0) years of clinical experience. The distribution of cases among physicians is given in Table 2.

In this high-difficulty, simulated primary care setting, real-time access to the AI diagnostic assistant was associated with higher Top-3 diagnostic accuracy than an unassisted, resource-restricted simulation condition. This finding was stable in leave-one-physician-out and virtual patient simulator exclusion analyses; the post hoc Top-2 analysis was supportive, whereas Top-1 was directionally favorable but statistically inconclusive. The study was designed as a diagnostic stress test rather than as an estimate of routine clinical effectiveness; therefore, absolute effect measures, including the simulation-context NNT-equivalent, should be interpreted within the curated challenging case mix.

The study also showed a possible safety-relevant overreliance signal when incorrect AI suggestions were visible, although the adjusted between condition comparison was imprecise and did not reach statistical significance. In the AI-assisted condition, concordance with an incorrect displayed suggestion can reasonably be interpreted as possible adoption of that suggestion. In the unassisted condition, however, concordance with a background AI suggestion reflects error concordance rather than true adoption, because physicians did not see the AI output.

Finally, exploratory analyses suggested marked heterogeneity, with benefit concentrated in more difficult cases and lower unassisted accuracy physician strata, while the easiest case quartile favored the unassisted condition. However, these subgroup findings should be considered hypothesis-generating because they were exploratory and partly derived from unassisted-condition performance.

These findings extend prior studies of LLM-based diagnostic support in 2 ways. First, most randomized evidence has evaluated static text vignettes or structured patient care tasks rather than real-time, voice-based support during an unfolding simulated consultation. In a vignette-based, randomized study, access to GPT-4 did not significantly improve physicians’ diagnostic reasoning compared with conventional resources, even though the standalone model performed well [9]. A later randomized trial of GPT-4 assistance in patient care tasks found a more modest physician-performance improvement in a sequential information environment [10]. Other recent work suggests that stronger benefits may occur when LLM support is paired with AI-literacy training or embedded in collaborative diagnostic workflows [11,12].

Second, this study evaluated an interactive voice-based workflow in which AI suggestions were generated during the consultation while physicians were still gathering information and forming hypotheses. This differs from benchmark studies showing strong standalone LLM performance on curated cases [7,8] and from evaluations focused mainly on passive documentation or operational efficiency. These results therefore support the view that the clinical value of diagnostic AI may depend not only on model capability but also on how model outputs are integrated into physicians’ real-time reasoning workflow.

The overreliance signal is also consistent with prior human-AI interaction research. Studies in digital pathology, radiology, and LLM-assisted diagnostic reasoning have shown that correct AI outputs can improve clinician performance, whereas incorrect outputs can degrade it [16,17,31]. More broadly, automation bias research has long shown that users may overweight automated recommendations in high-stakes environments, especially when systems appear authoritative or are presented as decision-support tools [32,33]. The higher crude concordance with incorrect suggestions in the AI-assisted condition was consistent with a possible overreliance concern in this real-time simulated primary care context, although the adjusted between-condition comparison was not statistically significant.

The main implication is not that real-time diagnostic AI assistance is ready for routine clinical deployment but that this type of system merits further formative refinement and prospective evaluation. In this controlled simulation, AI assistance appeared to support diagnostic performance. However, point estimates suggested a possible overreliance risk when the displayed suggestion was wrong, although the between-condition comparison was inconclusive. Future systems should therefore be evaluated not only for average diagnostic accuracy but also for how they behave under incorrect-suggestion scenarios and how physicians respond to those errors.

The results also suggest several design priorities. Interfaces should preserve physician verification rather than simply making AI suggestions more salient. Potential mitigation strategies include clearer uncertainty displays, prompts that ask clinicians to consider alternatives before accepting a suggestion, explicit flags when evidence is incomplete, and interface designs that separate hypothesis generation from recommendation endorsement. Training may also be important, because recent randomized evidence suggests that physicians may benefit more from LLM support when they receive AI-literacy training or when collaboration is structured through dedicated workflows [11,12].

Finally, the exploratory comparison with standalone AI should be interpreted cautiously. Neither physician-versus-AI contrast nor the human-by-arm interaction reached statistical significance, so these data do not establish additive human-AI synergy or a difference in relative performance between conditions. Whether specific workflows can produce reliable additive benefit beyond standalone model performance remains an empirical question for future studies.

The strengths of this study include case-level randomization, complete inclusion of all randomized consultations, blinded adjudication by 3 independent physicians, excellent interrater reliability, and mixed effects modeling with random intercepts for physician and case. The architecture also separated the patient simulator from the AI assistant, preventing the evaluated assistant from accessing the underlying vignette text or simulator prompts. The use of voice-based virtual consultations represents a more interactive evaluation setting than static text vignettes while retaining the safety and reproducibility advantages of simulation.

The study also has important limitations. First, this was a simulation-based physician performance study, not a clinical trial involving real patients or patient outcomes. Virtual patients cannot fully reproduce pain, anxiety, memory limitations, nonverbal cues, physical examination ambiguity, workflow interruptions, or the consequences of live clinical decisions. The retrospective assessment of the simulated patient performance showed imperfect behavior, which does not invalidate the findings, as demonstrated by the sensitivity analysis, but does underscore the inherently artificial nature of the environment.

Second, the comparator was an unassisted, resource-restricted simulation condition rather than resource-enabled usual care. Although this design standardized the experimental contrast, physicians in routine primary care may consult point-of-care references, clinical guidelines, internet searches, colleagues, or follow-up information. Access to such resources could improve unassisted performance and attenuate the observed relative and absolute contrast. However, the proportion of the observed effect attributable to the resource restriction cannot be estimated from this 2-condition design, because conventional resources could affect both unassisted and AI-assisted performance and may interact with AI use. Accordingly, the reported effect estimates apply specifically to AI assistance versus the resource-restricted comparator and should not be interpreted as the incremental effect of AI over resource-enabled usual care. Future studies should include a conventional resource comparator under the same time constraints and make the same resources available alongside AI assistance. This limitation is especially relevant because the case mix was intentionally difficult. As a result, the observed absolute accuracy gain and simulation-context NNT-equivalent may overestimate the absolute benefit that would be observed in routine care with easier cases and broader diagnostic resources.

Third, the physician sample was modest and recruited from a single Spanish regional health service. Although leave-one-physician-out analyses suggested that the primary result was not driven by a single physician, they address robustness to individual participants rather than external validity. The localized sample limits generalizability to physicians with different training backgrounds, experience, digital familiarity, health care systems, and practice environments. Because participation required use of a web application and voice interface, recruitment may also have favored physicians with greater baseline digital comfort. Future studies should recruit larger physician samples across multiple regions and health care systems and prospectively characterize participants’ digital literacy. The achieved allocation was also imbalanced because randomization was not blocked or stratified by physician or case, although the primary model included physician and case random intercepts and the leave-one-physician-out analyses supported robustness.

Fourth, several analyses were exploratory. Case difficulty and physician ability were derived from unassisted-condition performance, which introduces potential mathematical coupling when those metrics are reintroduced into the models. The workload, case position, physician ability, case difficulty, and standalone AI comparisons should therefore be treated as hypothesis-generating rather than confirmatory. The physician ability and safety models also yielded boundary estimates for the physician-level random effect variance, further supporting cautious interpretation.

Finally, the safety analysis used concordance with incorrect AI suggestions as an operational marker. In the AI-assisted condition, this is a plausible measure of possible overreliance because suggestions were visible. In the unassisted condition, however, the same measure represents background error concordance rather than true adoption. The adjusted 95% CI included no between-condition difference. Accordingly, the between-condition difference should be interpreted as a safety-relevant concordance signal rather than as a clinical harm estimate.

In this randomized simulation study, real-time, voice-based AI assistance was associated with higher Top-3 diagnostic accuracy than an unassisted, resource-restricted simulation condition. The study also identified a possible overreliance signal when incorrect AI suggestions were visible, although the between-condition safety estimate was imprecise and did not reach statistical significance. These findings support further refinement of diagnostic AI interfaces and prospective evaluation in real clinical workflows before any conclusions are drawn about routine clinical effectiveness or implementation.