This computational comparative study evaluated 4 LLMs: Claude Sonnet 4.5 (Anthropic), Gemini 2.5 Pro (Google), GPT-5 (OpenAI), and Grok 4 (xAI), accessed via their respective publicly available web interfaces in October 2025. Models were accessed using default parameters (temperature and other generation settings were not modified). Exact internal model checkpoints were not disclosed by model providers.

The primary objective was to comparatively evaluate LLM performance on anesthesiology examination questions. The primary end point was the percentage of correct responses across the 4 LLMs. Secondary objectives included assessment of hallucination rates and evaluation of question clarity through LLM-identified confusion. Secondary end points comprised the proportion of serious errors (hallucinations) and the proportion of questions flagged as “confusing” by the LLMs.

Questions were selected using a multidimensional approach drawing from (1) nurse anesthetist school examinations from Assistance Publique - Hôpitaux de Paris, designated as infirmier anesthésiste diplômé d’état (IADE; in English state-registered nurse anesthetist) questions; (2) European Diploma in Anaesthesiology and Intensive Care (EDAIC) examinations; and (3) EDAIC On-Line Assessment (OLA) practice examinations. All questions were available in online public repositories from prior examinations. None of the study authors were involved in the creation or selection of IADE examination questions.

The EDAIC, offered by the European Society of Anaesthesiology and Intensive Care, comprises written and oral components. The written examination includes 60 basic science multiple-choice questions (MCQs) followed by 60 clinical MCQs covering internal and emergency medicine; general and regional anesthesia; and specialized anesthesia, including pain management, resuscitation, and intensive care [13]. The OLA follows identical formatting for examination preparation. All questions were exclusively multiple choice to facilitate objective evaluation as MCQ responses are less subject to examiner subjectivity [14]. The IADE examination is the qualifying examination for French state-registered nurse anesthetists. The selected questions consisted of MCQs covering 5 domains (physiology, pharmacology, anesthetic techniques and implementation, procedure-related considerations, and patient-specific factors) and were administered in French, whereas EDAIC and OLA questions were administered in English. Questions were analyzed in their original language without translation.

Questions were submitted identically to the 4 LLMs following a standardized protocol. All models were queried using default web interface settings. Each question comprised a stem with 4 to 6 possible responses, with correct answers ranging from none to all options. A new conversation thread was initiated for each query to prevent cross-contamination of responses. Each individual answer option was scored as correct or incorrect, yielding a percentage success score for each question (ie, the proportion of correctly classified options out of all available options per question). This item-level scoring approach was chosen to capture partial knowledge and provide a more granular assessment of model performance than binary whole-question scoring. This scoring approach mirrors the official EDAIC examination marking scheme, in which each individual option is independently scored as correct or incorrect and partial credit is awarded accordingly.

The standardized prompt template was as follows: “[Question stem verbatim] Options: A) [option A] B) [option B] C) [option C] D) [option D] [E) and F) if applicable]. Please identify all correct answers and explain your reasoning.” This format was maintained consistently across all queries to ensure comparability.

All responses were evaluated through a structured consensus process by 2 examiners (TG and AB), who jointly reviewed each response, discussed discrepancies, and reached agreement on classification. When an LLM achieved less than 100% accuracy, joint review was conducted to distinguish between question-related confusion and true hallucination [7,10].

A hallucination was operationally defined as a response containing factually incorrect, fabricated, or unsupported medical information despite apparent comprehension of the question stem. In contrast, question-related confusion was defined as an incorrect response associated with misinterpretation of ambiguous or complex wording despite otherwise medically coherent reasoning. Representative examples illustrating these classification criteria are provided in Multimedia Appendix 1.

Hallucination and confusion analysis was performed for IADE and EDAIC questions only. The OLA was included as an external validation set for the primary end point (comparative success rates). Structured dual-reviewer classification of hallucinations and confusion for the OLA set would have required evaluation of 472 additional responses (118 × 4); the IADE and EDAIC provided 319 questions (1276 evaluated responses) across 2 distinct cognitive levels, which was considered sufficient to characterize error patterns. Given the fact that the OLA is not conceptually different from the EDAIC (it is an EDAIC simulation), it was retained as an external validation dataset for performance assessment rather than for detailed error pattern characterization.

Statistical analyses were performed using SPSS (version 29.0; IBM Corp) and Python (version 3.11; Python Software Foundation) with the statsmodels package (version 0.14), with the significance threshold fixed at a P value below .05 for all analyses. Quantitative variables are presented as means and SDs with 95% CIs where appropriate; qualitative variables are presented as counts and percentages. Normal distribution was assessed visually using histograms and Shapiro-Wilk testing. Results were analyzed by examination group (IADE, EDAIC, and OLA). Paired tests included Friedman and Wilcoxon signed-rank tests for quantitative variables and the Cochran Q test for qualitative variables. Pairwise post hoc comparisons following significant Friedman tests were performed using Wilcoxon signed-rank tests with exclusion of zero differences, and effect sizes were reported as r = Z/√n. To control the familywise error rate across the 6 pairwise comparisons per examination group while avoiding the excessive conservativeness of the Bonferroni method with correlated comparisons, the Holm-Bonferroni sequential procedure was applied. Holm-adjusted P values (denoted as “adjusted P”) are reported for all pairwise post hoc and domain-level comparisons and were compared against the fixed .05 significance threshold; unadjusted P values are reported only for omnibus tests, to which no multiplicity correction applies. The Kendall W was reported as the effect size for the Friedman and Cochran Q tests. To account for the clustered structure of the binary outcomes (each question was answered by the 4 models), pairwise intermodel comparisons of hallucination and confusion rates were performed using generalized estimating equations (GEE) with a logit link, a binomial distribution, an exchangeable working correlation structure, and robust SEs, with questions specified as clusters. All 6 pairwise model contrasts were estimated for each outcome, and the resulting P values were Holm adjusted.

This computational study did not involve human participants, patient data, biological material, or identifiable personal information. This study exclusively analyzed publicly available examination questions and AI-generated responses. According to French regulations governing biomedical research (Loi n° 2012-300 du 5 mars 2012, relative aux recherches impliquant la personne humaine, “Loi Jardé,“ modified by Ordonnance n° 2016-800 du 16 juin 2016 [15]), research not involving human participants, identifiable personal data, or biological samples falls outside the scope of mandatory ethics committee review. The Commission Nationale de l’Informatique et des Libertés (CNIL) similarly does not require declaration for research that does not process personal data. Ethics committee approval was therefore not required for the present study. No patient data were collected, stored, or analyzed at any stage of the study.

A total of 437 questions (1748 queries) were analyzed: 100 (22.9%) IADE questions, 219 (50.1%) EDAIC questions from 2 separate examinations, and 118 (27.0%) OLA questions. Complete results are shown in Tables 1-4 and Figure 1.

IADE performance results are summarized in Tables 1 and 2. Overall performance was high across all models, with no statistically significant intermodel differences observed (Friedman χ²₃=5.2; P=.16; W=0.02).

Questions were categorized into 5 domains: physiology (21/100, 21%), pharmacology (20/100, 20%), anesthetic techniques and implementation (14/100, 14%), procedure-related considerations (21/100, 21%), and patient-specific factors (24/100, 24%). After Holm adjustment across the 5 domain-level Friedman tests, no significant intermodel differences were observed (physiology: χ²₃=11.1, adjusted P=.06, and W=0.18; anesthetic techniques: χ²₃=8.3, adjusted P=.16, and W=0.20; pharmacology, procedure-related considerations, and patient-specific factors: adjusted P>.99). Given the small per-domain sample sizes, these analyses should be interpreted as exploratory.

EDAIC performance results are summarized in Table 1. Overall performance remained high across all models, with significant overall intermodel differences observed (Friedman χ²₃=13.9; P=.003; W=0.02). Claude Sonnet 4.5 and Gemini 2.5 Pro achieved the highest overall performances, whereas GPT-5 and Grok 4 showed lower mean scores.

In pairwise Wilcoxon signed-rank comparisons, Gemini 2.5 Pro significantly outperformed GPT-5 (Z=2.75; r=0.32; adjusted P=.04). No other pairwise comparisons were significant (adjusted P≥.09 in all cases).

OLA performance results are summarized in Table 1. Overall performance remained high across all models, with statistically significant overall intermodel differences observed (Friedman χ²₃=12.0; P=.008; W=0.03). Claude Sonnet 4.5 achieved the highest performance, whereas GPT-5 and Grok 4 showed lower mean scores. In pairwise Wilcoxon signed-rank comparisons, Claude Sonnet 4.5 significantly outperformed Grok 4 (Z=3.18; r=0.53; adjusted P=.009). No other pairwise comparisons were significant (adjusted P≥.08 in all cases).

Hallucination and confusion rates for each model are shown in Table 1. For IADE questions, hallucination rates did not differ significantly across models (Cochran Q₃=3.38; P=.34; W=0.01). The omnibus test for confusion rates reached nominal significance (Cochran Q₃=7.85; P=.049; W=0.03); however, the effect size was negligible, and no pairwise contrast was significant after Holm adjustment in the GEE analysis (Table 3; adjusted P≥.34 in all cases).

For EDAIC questions, neither hallucination rates (Cochran Q₃=6.22; P=.10; W=0.01) nor confusion rates (Cochran Q₃=6.28; P=.10; W=0.01) differed significantly across models. In the pairwise GEE analyses (Tables 3 and 4), no contrast was significant after Holm adjustment for either outcome; the largest contrast was observed for EDAIC hallucinations between Grok 4 and Gemini 2.5 Pro (odds ratio 1.52, 95% CI 1.08‐2.14; adjusted P=.09).

In this study, we compared the performance of 4 current-generation LLMs across multiple European anesthesiology examination datasets and evaluated hallucination and question-related confusion rates. All evaluated models demonstrated high overall performance, with success rates approaching or exceeding typical human passing thresholds for the EDAIC examination. Although overall intermodel differences were detected, a single pairwise difference per examination remained significant after correction, indicating that no model was consistently superior. Importantly, hallucination rates remained nonnegligible across all models, highlighting an important limitation for unsupervised educational use. Notably, LLM performance surpassed typical first-attempt pass scores for the EDAIC, which range between 60% and 75% (2025 passing scores: 67% for part A and 73.5% for part B), indicating that LLMs may outperform average examinees on standardized theoretical examinations [1,16].

The higher performance observed on IADE compared with EDAIC questions may reflect differences in cognitive complexity, as questions requiring lower-order cognitive skills (recall and comprehension) may be more amenable to current LLM capabilities than those requiring higher-order reasoning (analysis, synthesis, and evaluation) [5,17]. However, because IADE questions were administered exclusively in French whereas EDAIC and OLA questions were administered exclusively in English, the effects of examination type and language cannot be disentangled. Consequently, the observed performance differences may reflect differences in cognitive complexity, language, or examination format or a combination of these factors.

After Holm-Bonferroni correction, a single robust pairwise difference remained per examination: Gemini outperformed GPT-5 on the EDAIC, and Claude Sonnet 4.5 outperformed Grok 4 on the OLA. However, absolute performance differences between models were modest (2‐4 percentage points), suggesting that, while statistically detectable, these disparities may have limited practical significance. The clinically relevant finding is that all models performed comparably well, and hallucination rates represent a more meaningful differentiator for educational deployment.

These findings represent substantial improvement from previous reports. We evaluated GPT-3.5 in 2023 for the EDAIC part 1, finding a 70.5% success rate [1]. Other authors have documented similar performance with earlier model versions [18,19]. Our results demonstrate the rapid evolution of these technologies, consistent with other work showing progressive improvement across ChatGPT versions on medical licensing examinations worldwide [20]. In the United States, a recent exploratory study showed that ChatGPT can perform similarly to humans in standardized oral examinations as graded by American Board of Anesthesiology examiners [21], and other data show that current LLMs can surpass the minimum competency required for American Board of Anesthesiology certification [22]. Remarkable data have also emerged from various medical fields in recent years suggesting that LLMs can achieve high levels of performance in knowledge-based tasks while exhibiting important limitations in complex clinical reasoning and reliability [23,24].

Regarding hallucinations, our data do not support declaring any LLM superior. Confusion rates showed no significant intermodel differences except for a marginal effect on the IADE set (P=.049; W=0.03), which was of negligible magnitude and not corroborated by the pairwise GEE analysis. Training data may naturally contain errors that LLMs perpetuate, and users lack direct control over consulted sources, making verification challenging. Research on GPT-4o using Chilean national examination questions demonstrated that, despite strong factual recall and comprehension, the model exhibited limitations in complex reasoning and diagnostic judgment, producing unfounded medical assertions and imprecise conclusions [11]. Importantly, the hallucination rates of 11% (11/100) to 20.1% (44/219) in our study carry significant practical implications for medical education. A student relying on an LLM for self-directed learning may encounter erroneous information in approximately 1 out of every 5 to 10 queries, potentially reinforcing incorrect medical knowledge if not critically evaluated. Unlike factual errors that may be easily identified, LLM hallucinations are typically presented with high confidence and syntactic fluency, making them particularly insidious in an educational context. This underscores the necessity for supervised use and systematic cross-referencing with authoritative sources when LLMs are used as learning aids.

From an educational perspective, these findings suggest that LLMs should not be used as primary knowledge sources but rather as guided learning assistants. A pragmatic framework for safe integration may include (1) mandatory cross-verification with authoritative resources, (2) supervised use in structured teaching environments, and (3) explicit training of students to recognize and critically appraise AI-generated content. Without such safeguards, the observed hallucination rates could lead to reinforcement of incorrect medical knowledge during self-directed learning.

Our standardized prompt approach, while ensuring reproducibility, may not reflect optimal performance achievable through advanced prompting techniques. Chain-of-thought prompting, few-shot learning, or retrieval-augmented generation could improve accuracy and reduce hallucinations. Future studies should systematically evaluate the impact of prompt engineering on medical examination performance.

The high acceptance of AI tools among medical students further supports the relevance of these findings [25]. Surveys indicate widespread enthusiasm for AI integration into medical curricula, with students anticipating improvements in learning conditions and educational efficiency [26]. LLMs enable personalized educational approaches through tutorial dialogue simulation, complex concept summarization, and enhanced knowledge retention, positioning them as valuable complementary tools in medical education [20].

Beyond student support, these tools may reduce educator administrative burden by automating repetitive tasks, allowing more time for personalized student guidance and improving pedagogical quality [27,28]. The integration of LLMs into medical education raises important regulatory and policy considerations. While tools for self-directed learning may require less oversight than clinical decision support systems, educational institutions should develop clear policies regarding appropriate use, required supervision, and assessment of AI-generated content accuracy. Professional societies and accreditation bodies may need to establish guidelines for LLM use in medical education curricula.

Future studies examining open-ended questions or clinical cases would be valuable for testing LLM clinical reasoning capabilities. Given the exponential evolution of these technologies, our results will likely become obsolete within months, warranting repeated evaluation of subsequent model versions. Additionally, intervention studies evaluating the efficacy of LLM-assisted learning compared to traditional methods would help establish evidence-based recommendations for educational integration.

Several limitations warrant consideration. The consensus-based evaluation approach, while ensuring classification consistency between examiners, did not involve independent parallel scoring and, therefore, did not allow for formal quantification of interrater agreement through metrics such as the Cohen κ. Future studies should incorporate independent blinded evaluations to enable formal reliability assessment. Question difficulty was not formally graded. The evaluation relied on MCQs, which primarily assess knowledge recall and pattern recognition rather than complex clinical reasoning.

Questions originated from official institutions evaluating future professionals (EDAIC and IADE), and dual review reduced evaluation bias. While our substantial question sample merits expansion for enhanced statistical reliability, the exclusive use of MCQs (rather than open-ended questions) attenuated evaluation bias. Prompt formulation was standardized, with each LLM receiving identical questions in new conversation threads to prevent cross-contamination, although prior model exposure to these question banks cannot be excluded.

An important limitation is the potential presence of some examination questions or closely related content within LLM training datasets (“training data contamination”). Consequently, high performance may partially reflect memorization or prior exposure rather than genuine reasoning ability. However, several factors mitigate this concern: IADE questions were administered in French from an institutional repository with limited online dissemination, EDAIC questions are regularly updated by the examination committee, and the use of 3 distinct sources with different formats and languages reduces the likelihood of systematic contamination across all question sets. This concern has been increasingly acknowledged in the LLM benchmarking literature: prior work has demonstrated that training data contamination can artificially inflate performance on widely used medical examination benchmarks such as MedQA and US Medical Licensing Examination–style datasets, and dedicated detection methodologies have been proposed to estimate the extent of such contamination [29,30].

Additional methodological limitations include the fact that (1) questions were administered in their original language (French for the IADE and English for the EDAIC and OLA) and potential translation or language processing effects were not explored; (2) exact model version checkpoints were not documented beyond the October 2025 time frame, limiting precise reproducibility; (3) no test-retest reliability assessment was performed to evaluate response consistency—this single-query design was chosen to reflect typical real-world use patterns, where students or clinicians query an LLM once per question rather than repeatedly. However, future studies should assess response reproducibility across multiple runs.

Because the IADE questions were administered exclusively in French and the EDAIC and OLA questions were administered exclusively in English, language and examination type are inherently confounded in this design; consequently, the observed performance differences cannot be attributed to examination type alone and should be interpreted with this limitation in mind. Finally, there is a temporal limitation given the rapid LLM evolution as these results reflect October 2025 model versions.

Current-generation LLMs demonstrated high performance on anesthesiology examination questions. All models substantially exceeded human passing thresholds, consistent with previously reported progressive improvements in successive LLM iterations. However, hallucination rates remained persistent across the 4 models, emphasizing the importance of user awareness regarding this phenomenon. AI-generated outputs continue to require critical verification and supervised use in educational settings, but these models represent tools with substantial pedagogical and assessment potential. These findings highlight both the educational potential and the limitations of LLMs as supplementary learning tools for medical trainees preparing for standardized examinations. Educational deployment frameworks must therefore prioritize systematic verification mechanisms over reliance on model accuracy alone.