Agreement Between Reasoning-Oriented Generative AI Models and Clinical Educators in Evaluating Japanese Objective Structured Clinical Examination Transcripts: Preliminary Comparative Study

Introduction

Importance of Medical Interview Training

Medical interviewing is a cornerstone of modern health care, including medical education and practice [1-3]. Mastering clinical communication is not merely about gathering information; it is a critical bedside skill required to build rapport, accurately estimate pretest probability, and formulate differential diagnoses [4,5]. Accurate estimation of pretest probability is essential for guiding effective clinical investigations, thereby achieving diagnostic excellence [6-10]. Consequently, the ability to perform a structured and empathetic medical interview is a fundamental competency that medical learners must demonstrate before clinical practice [11].

Challenges in Traditional Educational Methods

Despite its importance, effective training in medical interviewing faces several challenges in implementation [12,13]. Unlike knowledge-based disciplines, which can be relatively easily scaled through mass lectures, digital media, or textbooks, clinical skills training is inherently resource-intensive [14]. Traditional methods, such as Objective Structured Clinical Examinations (OSCEs), require substantial investments in time, funding, and personnel, specifically due to the burden of recruiting and training standardized patients and securing clinical educators for supervision [15-20]. In resource-limited environments or institutions facing faculty shortages, providing medical learners with sufficient opportunities for medical interview training is often unfeasible [21]. Consequently, there is a growing disparity between the need for extensive clinical skills training and the educational infrastructure available to support it.

Furthermore, achieving objective and consistent evaluation remains a persistent difficulty, even within structured frameworks such as the OSCE. Although the OSCE was designed to standardize assessment through checklists and rating scales, human evaluation is inevitably subject to interrater variability and cognitive biases [22,23]. Studies have shown that interrater reliability can fluctuate due to factors such as variability in examiner stringency, known as the “hawk-dove” effect [24], and the “halo” effect, where an examiner’s general impression of a student influences checklist scores [25]. In the context of medical education, human evaluators often struggle to provide immediate and actionable feedback due to time constraints, limiting the educational value of the assessment for the learner [26].

Emergence of Artificial Intelligence in Medical Education

To address these systemic limitations, digital technology, including artificial intelligence (AI), has rapidly emerged as a transformative tool in health care, especially medical education [27-29]. Historically, rule-based AI applications were limited to rigid simulations or diagnostic algorithms [30]. However, recent advancements have expanded the utility of AI to include personalized learning assistants, virtual patient simulations, and automated feedback systems [31].

Generative Artificial Intelligence and Natural Language Processing

Generative artificial intelligence (GenAI), a subfield of AI that uses natural language processing, holds promise for medical education [32,33]. These systems are powered by large language models (LLMs), which process and generate human-like text by predicting word sequences based on probabilistic models trained on massive datasets [34].

Unlike earlier rule-based AI, modern LLMs can parse complex, unstructured data—such as the dialogue of a medical interview—and generate contextually relevant responses or analyses [35,36]. Two of the most prominent state-of-the-art generative models currently available are generative pretrained transformer (GPT) and Gemini, developed by OpenAI and Google, respectively [37-39]. These general-purpose, unspecialized platforms have demonstrated remarkable capabilities in medical reasoning and performance on licensing examinations [40-42]. However, their utility is not limited to answering questions; they possess the potential to act as evaluators, assessing medical learners’ performance against established clinical criteria.

Previous Work

In a preceding investigation, we explored the feasibility of using the previous GenAI model, GPT-4 (OpenAI), as a supplemental evaluation tool for Japanese OSCEs [43]. This experimental study compared GPT-4’s evaluation of medical students against the consensus of experienced physicians. The results highlighted a dichotomy in performance: while GPT-4 provided evaluation scores comparable to clinical educator consensus ratings in the domains of patient care or communication and history taking, the GenAI model overestimated student performance in physical examination, clinical reasoning, and patient notes compared to the physicians. Furthermore, the agreement between the GenAI and the clinical educator consensus ratings, as measured by intraclass correlation coefficients (ICCs), remained low across most domains. These findings suggested that while GenAI possesses the potential to supplement medical education, particularly in communication-based tasks, its reliability as a standalone evaluator remains unproven, and tendencies toward score inflation require attention.

Reasoning-Oriented LLMs

Furthermore, the latest evolution in LLMs introduces reasoning capabilities, often referred to as chain-of-thought processing [44]. Unlike standard models that predict the next word based on probability, recent GenAI platforms include reasoning-oriented models designed to improve performance on tasks requiring multistep reasoning [45]. Although such systems may generate outputs consistent with more extensive reasoning processes, their internal reasoning mechanisms were not directly accessible or analyzed in the present study. The impact of reasoning-oriented model design on the subjective evaluation of medical trainees, therefore, remains uncertain.

Gaps and Study Aim

While the capabilities of reasoning-oriented GenAI models are well-documented, their validity as evaluators of clinical skills remains underexplored, particularly in non-English contexts [46,47]. The nuances of the Japanese language—which relies heavily on context, honorifics, and indirect communication—pose unique challenges for LLMs compared to English [48]. Given the contextual complexity of the Japanese language, it is currently unknown whether general-purpose GenAI models such as GPT and Gemini, especially reasoning-oriented models, can accurately evaluate Japanese medical interviews with a level of reliability comparable to that of clinical educators.

Therefore, this preliminary study aims to assess the agreement between reasoning-oriented GenAI models and clinical educator consensus ratings in the evaluation of Japanese OSCEs. Specifically, we aim to compare the evaluation scores generated by these GenAI models against those provided by experienced clinical educators to determine if GenAI can serve as a reliable, resource-efficient adjunct in medical training. Demonstrating alignment with human experts would provide preliminary support for these tools as scalable adjuncts to address faculty limitations and standardize assessment.

Methods

Setting

This study was conducted as a preliminary comparative analysis within the Department of Diagnostic and Generalist Medicine (general internal medicine) at Dokkyo Medical University, Tochigi, Japan.

The current research used a dataset derived from our prior investigation into the utility of GenAI for Japanese medical interview training [49]. This study consisted of three phases. First, we prepared text data from medical interview training sessions. Second, human clinical educators evaluated these data. Third, GenAI models evaluated the same text data. The entire study, including all medical interview training, transcripts, and evaluations, was conducted in Japanese; no translated versions were used. Both GenAI models evaluated the original Japanese-language transcripts directly; no English translations or other translated versions were used for model evaluation. The flowchart, including participants, group allocation, and evaluation by clinical educators and both GenAI models, is shown in Figure 1.

The University Hospital Medical Information Network registration refers to the original medical interview training study from which the transcript dataset was derived. The present comparative evaluation of GenAI-generated and clinical educator scores was conducted as a secondary analysis of those previously collected transcripts and was not separately prospectively registered.

‎

Ethical Considerations

This study protocol was reviewed and approved by the Institutional Review Board of the Dokkyo Medical University Hospital (No. R-79‐14J). All participants provided written informed consent. No additional platform-level data retention settings were available or verified through the interfaces used at the time of evaluation. Only the anonymized text required for evaluation was submitted.

Text Data From Medical Interview Training

We recruited postgraduate year 1 (PGY-1) and postgraduate year 2 (PGY-2) residents rotating through the general internal medicine department. Participants were randomized to a crossover sequence alternating between two distinct training styles to prepare this study’s data: (1) a text-based medical interview training modality using a custom GPT chatbot to simulate a patient, hereafter referred to as the chatbot-based training style, and (2) a traditional face-to-face medical interview training with a human standardized patient, referred to as the traditional training style. Subsequently, the text data derived from both training styles were anonymized and independently evaluated by three evaluator sources: the clinical educator consensus ratings, GPT-5.2 Thinking, and Gemini 3.0 Pro. The GPTs, developed by OpenAI, were custom GPT-based chatbots [50], configured via a noncoding interface to simulate specific patient personas.

Each participant completed 2 stations. The first station featured a case of abdominal pain, and the second station featured a case of chest pain. Both training styles, chatbot-based and traditional, were applied to these identical clinical cases. To ensure consistency, participants in both styles were only required to state or type the specific physical examination maneuvers they intended to perform. Following the medical interview part, the participants were required to present their clinical assessment and plan for each case. All training sessions were video-recorded to facilitate transcription. The detailed protocols for the medical interview training are provided in Multimedia Appendix 1. Summaries of the two clinical cases used in medical interview training are shown in Multimedia Appendix 2.

No formal sample size calculation was performed because this was a preliminary exploratory study using available transcripts from a medical interview training program. The sample size was determined by the number of participating residents and completed training transcripts during this study’s period.

Data Extraction and Preparation

Following the medical interview training sessions, the content of all medical interviews and subsequent case presentations was extracted as text data for analysis. For the chatbot-based sessions, text data were generated by directly extracting the conversation logs. For the traditional sessions, the dialogue was manually transcribed by the primary researcher (TH) from the video recordings. To ensure the evaluations focused solely on content, the text data were formatted to remove identifiers of the modality used, including specific formatting unique to the chatbot interface. Although identifiers of the training modality and formatting elements unique to the chatbot interface were removed before evaluation, residual differences in transcript length, verbosity, interaction structure, completeness, or transcription characteristics could remain because chatbot-based records were extracted directly from conversation logs, whereas traditional sessions were manually transcribed from video recordings. As an exploratory analysis, we compared transcript length, measured as the total number of Japanese characters, between the two training modalities using the paired Wilcoxon signed-rank test. An example of anonymized transcription formatting is provided in Textbox 1.

Evaluation by Clinical Educators

Two independent clinical educators (MY and T Sakamoto) evaluated the anonymized text data. The evaluation criteria were based on standard OSCE scoring rubrics with a 6-point Likert scale, where 1 is inferior and 6 is excellent [51,52]. The assessment covered six specific domains: (1) patient care and communication skills, (2) thoroughness of history-taking, (3) physical examination proficiency, (4) accuracy and organization of clinical information (evaluated for the logical flow and organization of the clinical facts presented, rather than formal medical recording standards), (5) clinical reasoning capability, and (6) overall patient management strategies. Crucially, both evaluators were blinded to the training modality, chatbot-based vs traditional, corresponding to each transcript. The clinical educators evaluated the same text-only anonymized transcripts that were provided to the GenAI models and did not have access to video recordings, audio recordings, training modality information, participant identities, or any additional contextual information from the training sessions. The two clinical educators first completed their evaluations independently and without access to one another’s scores. Their original independent ratings were preserved before consensus discussion and were used for the preconsensus interrater agreement analysis. For transcripts with discrepant scores, the educators reviewed the six rubric domains and discussed the relevant transcript content with reference to the predefined rubric descriptors. A consensus score was then assigned for each domain. No third adjudicator was used because all discrepancies were resolved through discussion between the two educators. To control potential order effects, the sequence of evaluations was randomized using a randomization list generated via Microsoft Excel by another researcher (KM). The detailed evaluation rubric for medical interview training is presented in Multimedia Appendix 3.

Evaluation by GenAI

Overview

For GenAI evaluation, we selected two state-of-the-art reasoning-oriented GenAI models: GPT-5.2 Thinking, developed by OpenAI, and Gemini 3.0 Pro, developed by Google LLC. These models were selected for their advanced reasoning-oriented capabilities and accessibility.

Both models were tasked with evaluating the entire dataset of interview and presentation transcripts. The same anonymized text data evaluated by the clinical educators was fed into both GenAI models. Each evaluation was based on a single generated output per transcript by each model. Repeated evaluations, alternative prompts, and parameter-tuning experiments were not performed in the primary analysis. No worked scoring examples, educator-approved anchor cases, few-shot examples, or model-specific calibration materials were supplied to either GenAI model. To prevent context retention between cases, the model context was refreshed after every individual evaluation session. The specific prompt engineering used to guide the GenAI models’ evaluations was standardized as follows in Textbox 2. The GenAI models were instructed to provide numerical scores for each rubric domain. Qualitative feedback or narrative comments were not requested, collected, or analyzed in this study. GPT-5.2 Thinking was accessed through the ChatGPT (OpenAI) web interface. Gemini 3.0 Pro was accessed via Google AI Studio. All GenAI-based evaluations were executed on January 9, 2026.

Available settings were kept at the platform defaults. For GPT-5.2 Thinking, model parameters such as temperature, top-p, and output length were not visible or adjustable in the ChatGPT web interface used in this study; therefore, the platform default settings were used. For Gemini 3.0 Pro, the following Google AI Studio settings were used: temperature, 1.0 on a scale from 0 to 2; thinking level, high, selected from low, medium, and high; top-p, 0.95 on a scale from 0 to 1; and output length, 65,536 tokens. No additional user-defined system-level instructions were used beyond the standardized evaluation prompt.

Data Collection and Outcomes

We collected background data, including participants’ biological sex and their years since obtaining a medical degree. Additionally, the length of each medical interview transcript was recorded as the total number of Japanese characters in the original Japanese transcript.

The primary outcome was the difference in the overall OSCE evaluation score among the three evaluator groups: clinical educator consensus score, GPT-5.2 Thinking, and Gemini 3.0 Pro. The overall score was defined as the mean score across the six rubric domains. Pairwise comparisons of the overall score were performed among the three evaluator groups. The secondary outcomes included domain-level score differences across the six rubric domains and interrater agreement between each GenAI model and the clinical educator consensus score, assessed using ICCs. Domain-level score comparisons were considered exploratory.

Additionally, as an exploratory outcome, we assessed the intermodel reliability (agreement between GPT-5.2 Thinking and Gemini 3.0 Pro). Furthermore, we investigated the influence of interaction modality by comparing overall scores and agreement specifically between the chatbot-based and the traditional styles.

Statistical Analysis

Evaluation scores are presented as means with 95% CIs. Differences in scores between groups (clinical educator consensus score vs GPT-5.2 Thinking and clinical educator consensus score vs Gemini 3.0 Pro) were analyzed using the Wilcoxon signed-rank test. To better reflect the ordinal nature of the rubric, medians and IQRs are also reported in Multimedia Appendix 4.

To characterize the reference standard, we assessed preconsensus agreement between the two clinical educators using ICCs based on a two-way random-effects model for absolute agreement and single measurements. We also calculated exact agreement, agreement within 1 point, mean absolute difference, maximum absolute difference, and mean difference. These metrics were calculated by domain and for the overall score.

Agreement between evaluators was assessed using ICCs calculated with a two-way random-effects model for absolute agreement and single measurements. This model treats both participants and raters as random effects and evaluates whether individual ratings from each GenAI model agree with those of the human clinical educators. ICC estimates are reported together with their corresponding 95% CIs. The degree of reliability was interpreted according to the following guidelines: ICC values less than 0.5 indicate poor reliability; values between 0.5 and 0.75 indicate moderate reliability; values between 0.75 and 0.9 indicate good reliability; and values greater than 0.9 indicate excellent reliability [53].

As each resident completed two stations, the 40 transcript-level observations were not fully independent. To address this repeated structure, we conducted a participant-level sensitivity analysis for the primary outcome. For each resident, the overall scores from the two stations were averaged within each evaluator group, yielding one person-level overall score per resident for clinical educators, GPT-5.2 Thinking, and Gemini 3.0 Pro. We then repeated the paired comparisons using Wilcoxon signed-rank tests at the participant level (n=20).

We did not fit a mixed-effects model in the present preliminary analysis because the dataset included only 20 residents and two stations representing two clinical cases. With only two case levels and a modest sample size, simultaneous estimation of participant-level and case-level random effects would have limited stability and interpretability. Future studies with larger numbers of participants and stations should use mixed-effects models to account for resident-level clustering and case effects.

Given the multiple comparisons performed in this study, we applied the Bonferroni correction to adjust the significance levels and control the family-wise error rate [54]. For the primary confirmatory analysis, Bonferroni correction was applied to the three pairwise comparisons of the overall OSCE score among the evaluator groups: GPT-5.2 Thinking vs clinical educator consensus score, Gemini 3.0 Pro vs clinical educator consensus score, and GPT-5.2 Thinking vs Gemini 3.0 Pro. Accordingly, the Bonferroni-corrected significance threshold for the primary analysis was P<.017 (.05/3). Domain-level score comparisons across the six rubric domains were conducted as secondary exploratory analyses and are reported with exact P values; these analyses were not included in the primary Bonferroni correction. ICC analyses were interpreted descriptively using established reliability thresholds and were not subjected to Bonferroni correction. All statistical analyses were performed using R software (version 4.2.2; R Foundation for Statistical Computing).

Results

Characteristics

A total of 20 physicians participated in this study. Among participants, 55% (11/20) were PGY-1 and 45% (9/20) were PGY-2. Further, 90% (18/20) of participants were male, indicating a predominantly male sample. This sex imbalance should be considered when interpreting the generalizability of the findings to more sex-balanced resident cohorts.

Regarding text data, a total of 40 medical interview transcripts were analyzed, including 20 chatbot-based transcripts and 20 traditional-style transcripts. Clinical educators and both GenAI models rated the entire dataset of 40 transcripts. Japanese character count ranged from 872 to 5167 (mean 1769, 95% CI 1463 to 2074).

Evaluation Scores: Comparison Between Reasoning-Oriented GenAI Models and Clinical Educators

Table 1 summarizes the comparison of scores assigned by GPT-5.2 Thinking, Gemini 3.0 Pro, and the clinical educator consensus ratings. Across all domains, the clinical educator consensus ratings were higher than the GenAI model scores.

For the primary outcome, the clinical educator consensus mean score was 5.18 (95% CI 5.06 to 5.30). In contrast, Gemini 3.0 Pro assigned a significantly lower mean score of 4.09 (95% CI 3.97 to 4.21; P<.001), and GPT-5.2 Thinking assigned the lowest score of 3.68 (95% CI 3.62 to 3.72; P<.001). The median overall score was 5 (IQR 5‐6) for the clinical educator consensus rating, 4 (IQR 3‐4) for GPT-5.2 Thinking, and 4 (IQR 3‐5) for Gemini 3.0 Pro. Median and IQR values for each evaluator group and domain are detailed in Multimedia Appendix 4.

This trend of lower GenAI scores persisted across every subdomain. The discrepancy was most pronounced in the management domain, where the clinical educator consensus mean score was 5.20 (95% CI 4.91 to 5.49), whereas GPT-5.2 Thinking gave a score of 2.93 (95% CI 2.79 to 3.06), and Gemini 3.0 Pro gave 3.58 (95% CI 3.28 to 3.87). Similarly, in physical examination, human evaluators gave high marks (mean 5.45, 95% CI 5.22 to 5.68), while both GPT-5.2 Thinking (mean 3.90, 95% CI 3.74 to 4.06) and Gemini 3.0 Pro (mean 4.08, 95% CI 3.70 to 4.45) rated the performance significantly lower.

As each resident contributed two transcripts, we conducted a participant-level sensitivity analysis using the mean overall score across the two stations for each resident. The findings were consistent with the transcript-level analysis. At the participant level, clinical educator consensus ratings yielded the highest mean overall score (5.18, 95% CI 4.99 to 5.37), followed by Gemini 3.0 Pro (4.09, 95% CI 3.91 to 4.28) and GPT-5.2 Thinking (3.68, 95% CI 3.59 to 3.76). Paired participant-level comparisons showed that GPT-5.2 Thinking scored lower than the clinical educator consensus score (mean difference −1.50, 95% CI −1.68 to −1.32; P<.001), Gemini 3.0 Pro scored lower than the clinical educator consensus score (mean difference −1.09, 95% CI −1.26 to −0.91; P<.001), and Gemini 3.0 Pro scored higher than GPT-5.2 Thinking (mean difference 0.42, 95% CI 0.25 to 0.58; P<.001). These results were consistent with the primary transcript-level analysis after participant-level aggregation. Participant-level sensitivity analysis of overall scores is provided in Multimedia Appendix 5.

Preconsensus Agreement Between Human Clinical Educators

As shown in Table 2, agreement between the two clinical educators was poor. For the overall score, the preconsensus ICC was 0.14 (95% CI 0.05 to 0.23). Exact agreement for the overall score was 0.0%, and agreement within 1 point was 47.5%. The mean absolute difference in overall score was 1.13 points, with a maximum absolute difference of 2 points. The mean signed difference was 1.12 points, indicating that one educator tended to assign higher overall scores than the other educator before the consensus discussion.

These findings indicate that substantial interhuman variability existed before consensus-building. Therefore, the consensus score was used as a practical reference standard, but poor GenAI-human agreement should be interpreted in the context of preconsensus human rater variability and possible rubric subjectivity.

Interrater Agreement

As shown in Table 3, the agreement between the GenAI models and the clinical educator consensus ratings was generally poor across all domains. The ICCs between GPT-5.2 Thinking and clinical educator consensus score were extremely low, ranging from 0.02 in management to 0.11 in accuracy and organization of information, with an overall agreement of 0.04 (95% CI 0.00 to 0.09), indicating poor reliability.

The agreement between Gemini 3.0 Pro and the clinical educator consensus score was slightly higher but still classified as poor. The overall ICC between Gemini 3.0 Pro and clinical educator consensus score was 0.22 (95% CI 0.10 to 0.35). The highest agreement for Gemini was observed in history taking (ICC 0.28) and the lowest in management (ICC 0.14).

As an additional post hoc exploratory analysis, we examined calibration-related characteristics of the GenAI model scores relative to the clinical educator consensus ratings. The mean signed difference was −1.50 points for GPT-5.2 Thinking and −1.09 points for Gemini 3.0 Pro. For the overall score, GPT-5.2 Thinking showed a Spearman rank correlation of 0.31 and an ordering concordance of 48.9%, whereas Gemini 3.0 Pro showed a Spearman rank correlation of 0.62 and an ordering concordance of 69.2%. These findings suggest that the disagreement was not explained solely by a constant additive score offset. A post hoc exploratory calibration analysis and three retrospective illustrative examples are provided in Multimedia Appendix 6.

Exploratory Analysis

When comparing the two reasoning-oriented GenAI models from Table 1, Gemini 3.0 Pro generally assigned higher scores than GPT-5.2 Thinking. These differences were statistically significant in the domains of patient care (P=.01), history taking (P<.001), clinical reasoning (P<.001), and management (P<.001). However, there was no significant difference between the two GenAI models in the scores for physical examination (P=.38) and accuracy and organization of clinical information (P=.14).

As shown in Table 3, agreement between the two GenAI models (GPT-5.2 Thinking vs Gemini 3.0 Pro) was also poor, with an overall ICC of 0.28 (95% CI 0.10 to 0.43). The highest agreement between the models was found in history taking (ICC 0.35) and clinical reasoning (ICC 0.31).

Table 4 presents the evaluation scores classified by training style, chatbot-based vs traditional. For GPT-5.2 Thinking, no statistically significant difference in overall scores was detected between the chatbot-based and traditional styles (P=.44). For Gemini 3.0 Pro and clinical educator consensus score, mean scores were higher in the traditional style, but these differences did not reach statistical significance after Bonferroni correction (P=.03 for both; corrected threshold P<.017). Therefore, no statistically significant modality-related difference in overall scores was detected after correction. However, because this exploratory comparison involved a modest sample size and was not designed as an equivalence or noninferiority analysis, these findings should not be interpreted as evidence that assessment performance was robust or equivalent across training modalities.

Transcript length differed between the two training modalities. The median Japanese character count was 1754.5 (IQR 1395.3‐2558.3) for the traditional style and 1280.5 (IQR 1141‐1527) for the chatbot-based style. Traditional-style transcripts were significantly longer than chatbot-based transcripts (P=.001). Additional exploratory analyses of the associations between Japanese character count, evaluator scores, and absolute GenAI–clinical educator consensus score discrepancies are provided in Multimedia Appendix 7.

Discussion

Principal Results

This preliminary study evaluated the agreement between two reasoning-oriented GenAI models (GPT-5.2 Thinking and Gemini 3.0 Pro) and clinical educator consensus ratings when assessing Japanese medical interview training. The investigation yielded four critical findings regarding the comparative behavior of GenAI models vs human evaluation.

First, we observed systematically lower score assignment and poor relative agreement in both reasoning-oriented GenAI models. Clinical educator consensus ratings were generally high, mean scores exceeding 5 on a 6-point scale, reflecting a supportive, competency-based approach. This divergence was most profound in the management domain, where the mean score gap reached nearly 2.3 points between GPT-5.2 Thinking (mean 2.93) and clinical educator consensus score (mean 5.20). One possible explanation is that the models interpreted the rubric more literally or applied different scoring thresholds than the clinical educators. For example, clinical educators may have applied a pragmatic supervised-practice standard appropriate for PGY-1 and PGY-2 residents, whereas the models may have required more explicit documentation of comprehensive performance. However, this interpretation remains a hypothesis because model rationales were not collected, and item-level qualitative error analysis was not performed.

Second, this study highlights differences between the reasoning-oriented GenAI models. Gemini 3.0 Pro assigned higher scores than GPT-5.2 Thinking under the tested conditions. Specifically, Gemini 3.0 Pro consistently assigned significantly higher scores than GPT-5.2 Thinking across complex domains such as clinical reasoning and history taking. This observed difference suggests that the two platforms applied different scoring thresholds under the tested conditions. This study cannot determine whether the difference arose from model architecture, training data, alignment strategies, platform settings, or other factors.

Third, the reliability of these models for human assessment remains poor. The ICCs between each GenAI model and clinical educator consensus score were consistently below 0.5 across all domains, with agreement for GPT-5.2 Thinking being negligible (ICC=0.04). The low ICC between each GenAI and clinical educator consensus score indicates that the models ranked resident performance differently from the educator consensus ratings under the tested conditions. Consequently, the current unspecialized, untrained reasoning-oriented GenAI models should not be used as standalone alternatives under the tested conditions.

Importantly, the clinical educator consensus ratings were not an error-free gold standard. The two educators showed poor preconsensus agreement, indicating substantial interhuman variability and possible subjectivity in rubric application. Therefore, poor GenAI–human agreement may reflect both differences in model scoring behavior and instability in the human reference standard. Future validation studies should include larger educator panels, structured calibration exercises, and predefined anchor examples.

A retrospective review of three illustrative cases provided additional context for this hypothesis. In two transcripts with high clinical educator consensus ratings, the summarized assessment and plan were concise, but the clinical educators judged that the participants had appropriately addressed potentially life-threatening conditions and demonstrated performance suitable for supervised practice. The GenAI models assigned lower scores, particularly for clinical reasoning and management. In another relatively brief transcript, both the clinical educators and GenAI models assigned comparatively lower scores. These examples suggest that the degree of explicit documentation may influence scoring differently across evaluators. However, because the examples were selected retrospectively and model rationales were not collected, this interpretation remains hypothesis-generating.

Fourth, the exploratory analysis examined whether overall evaluation scores differed by training modality. No statistically significant difference was detected after correction between the chatbot-based and traditional training styles for any evaluator group. However, this finding should be interpreted cautiously because the subgroup comparison was exploratory and may have been underpowered. A lack of statistically significant difference does not establish equivalence or robustness across modalities. Larger studies specifically designed to evaluate modality effects are needed before drawing firm conclusions about whether transcript-based assessment is unaffected by training modality.

Possible Explanations for Disagreement Between GenAI Models and Clinical Educators

Several mechanisms may explain the poor agreement between the GenAI models and the clinical educator consensus ratings. First, the GenAI models may have interpreted the rubric more literally than clinical educators. In actual OSCE evaluation, clinical educators may apply holistic judgment that considers the learner’s postgraduate level, overall safety, clinical plausibility, and adequacy for supervised practice. In contrast, the models may have treated each rubric descriptor as requiring comprehensive or textbook-level performance, thereby assigning lower scores when transcripts lacked explicit details.

Second, we did not perform a formal item-level qualitative error analysis of individual cases or model-generated rationales. Future studies should examine cases with large human-GenAI score discrepancies, classify the sources of disagreement, and determine whether calibration examples, few-shot prompting, or human-labeled training data can reduce systematic bias and improve alignment with clinical educator consensus ratings.

Strengths of this Study

This study has several strengths. First, the inclusion of two reasoning-oriented GenAI platforms allows for a comparative assessment of platform-specific scoring behavior, showing that Gemini 3.0 Pro assigned higher scores than GPT-5.2 Thinking under the tested conditions. Second, we directly compared the evaluation performance of these reasoning-oriented models within a non-English context. The use of Japanese text data adds value by examining model evaluation behavior in a non-English, high-context linguistic setting, although language-specific mechanisms were not directly analyzed. Finally, the same anonymized text data evaluated by the clinical educators were fed into both reasoning-oriented GenAI models. This was complemented by a blind evaluation process for the clinical educators, minimizing potential bias.

Limitations

Several limitations should be considered when interpreting these findings. First, this study was conducted using transcripts exclusively in the Japanese language and within a single institutional training context characterized by a predominantly male cohort (18/20, 90%). This demographic imbalance, combined with the single-center design, may limit generalizability to other languages, educational systems, and clinical environments. Although this study focused on Japanese-language transcripts, we did not directly analyze how Japanese-specific linguistic features influenced model scoring. Future studies should compare model performance across Japanese originals, English translations, and bilingual human ratings, and should examine whether specific Japanese communication features contribute to scoring disagreement.

Second, a central methodological limitation is that this study evaluated a single operational setup: one output per transcript, generated through zero-shot prompting without educator-approved calibration examples or rubric anchors. Repeated evaluations, prompt-sensitivity analyses, few-shot calibration, and systematic parameter comparisons were not performed [44,55]. Accordingly, the results do not establish the intrinsic capability of either model across alternative configurations. Specifically, only two reasoning-oriented GenAI models were evaluated at a single time point, and model performance may change substantially with future updates, parameter tuning, or alternative prompt engineering strategies. Consequently, the performance of other LLMs, such as Claude, developed by Anthropic, Llama, developed by Meta, and DeepSeek, by Hangzhou DeepSeek Artificial Intelligence Co, Ltd, remains unknown [56]. Within the OpenAI ecosystem, we did not evaluate the GPT-5.2 Pro model or ChatGPT Health (OpenAI) [57,58]. We used zero-shot prompting to ensure methodological consistency with the human evaluators. As LLM outputs may vary across multiple outputs, prompts, parameter settings, and model versions, this study could not estimate intramodel consistency or prompt sensitivity. Repeated evaluations using the same model versions would have strengthened the reliability assessment; however, the exact model versions used in the original evaluation were no longer accessible after subsequent platform updates. This limitation directly affects the interpretation of the reliability findings, as poor agreement with human educators may reflect not only systematic differences in scoring thresholds but also run-to-run variability. To address this, future research should incorporate one-shot or few-shot learning techniques [59,60].

Additionally, although standardized prompts were used and model context was reset between evaluations, stochastic variability inherent to LLMs may still influence scoring reproducibility. As only one output was generated per transcript, we could not quantify run-to-run variability [61]. Furthermore, the practical reference standard relied on consensus ratings from two clinical educators rather than a larger panel, which may limit the precision of the reference standard and restrict estimation of interhuman variability [62]. Although explicit modality identifiers and chatbot-specific formatting were removed, residual differences in transcript length, verbosity, dialogue structure, completeness, or transcription characteristics may have influenced the evaluation scores. Moreover, the evaluation was based solely on textual transcripts, excluding real physical examination, nonverbal communication cues, tone, and real-time interaction dynamics that are integral to authentic clinical assessment and may differentially influence human and GenAI judgment. In addition, the sample size was modest and included only two clinical scenarios. Therefore, this study may have limited statistical power for domain-specific analyses, including management-domain findings. The results may also be influenced by case specificity and may not generalize to other OSCE cases, learner levels, institutions, or clinical contexts. Although the participant-level sensitivity analysis addressed clustering by resident, this study still included only two clinical cases, and case-specific effects could not be fully separated from evaluator effects. Future studies with larger numbers of stations should use mixed-effects models to account simultaneously for participant, case, and evaluator-level variability. Finally, the reproducibility and governance of commercial GenAI-platform use depend partly on interface-level settings and platform policies, some of which may not be fully visible or independently verifiable to researchers.

Transcript length may also have influenced the evaluation results. Longer transcripts were associated with higher clinical educator consensus ratings and larger absolute discrepancies between the clinical educator consensus ratings and both GenAI model scores. However, these exploratory associations do not establish causality. Transcript length may reflect training modality, verbosity, transcript completeness, interaction structure, case-specific characteristics, or the quality of resident performance.

Comparison With Prior Work

A critical insight emerges when comparing these results to our previous investigation using GPT-4 [43]. In the prior study, GPT-4 exhibited a tendency toward score inflation, assigning significantly higher scores than physicians in domains such as clinical reasoning (median 5.0, IQR 5.0-5.0 vs median 4.0, IQR 3.0-4.0; P<.001), and management (median 6.0, IQR 5.0-6.0 vs median 4.0, IQR 2.5-4.5; P=.002).

In contrast, this study showed a scoring tendency in the opposite direction, with lower scores assigned by GPT-5.2 Thinking and Gemini 3.0 Pro than by the clinical educator consensus ratings. As the prior and current studies differed in datasets, learner populations, rubrics, prompts, platforms, and experimental conditions, the observed difference in scoring direction cannot be attributed to model evolution alone. For example, in the management domain, where GPT-4 previously overestimated performance, GPT-5.2 Thinking underestimated it (mean 2.93 vs human 5.20). Despite this reversal in scoring direction, the lack of reliability remains consistent between the two studies. Both the previous study (ICC between GPT-4 and clinical educator consensus score for total score=0.00) and the current study (ICC between GPT-5.2 Thinking and clinical educator consensus score for overall score=0.04) show that the evaluated configurations did not rank residents consistently with educator consensus ratings. This scoring direction may reflect differences in model architecture, training, alignment strategies, prompt, rubric interpretation, or other experimental conditions. However, this study cannot determine the mechanism underlying this change. The observed score deflation may indicate that these models applied stricter or more literal rubric interpretation under the tested prompt conditions, but the present study cannot determine the underlying mechanism [63].

Future Directions

Future research should evaluate larger and more diverse datasets across multiple institutions, languages, and clinical scenarios to assess the robustness and generalizability of GenAI-based evaluation. Incorporating multimodal inputs, such as audio, video, and nonverbal behavior, may improve model alignment with human assessment in communication and professionalism domains [64]. Methodologically, ensemble approaches combining multiple models, calibration techniques, or rubric anchoring using human-labeled training data may reduce systematic bias and improve agreement [65]. Furthermore, future investigations should use mixed methods designs, combining quantitative scoring analysis with qualitative thematic analysis of GenAI-generated feedback.

Future studies should separately evaluate representative educational use cases: summative scoring, formative narrative feedback, calibration support for clinical educators, and learner self-reflection. Evidence supporting one use case should not be assumed to establish validity for the others. As this study collected and analyzed only numerical scores, it cannot determine whether the models’ qualitative feedback would be accurate, educationally useful, or acceptable to learners and educators. Future studies should explicitly instruct models to generate narrative feedback, collect those outputs systematically, and evaluate them using qualitative thematic analysis, educator review, and learner-centered outcomes. Finally, integrating explainability mechanisms that provide transparent rationales for scores may enhance educator trust, facilitate error analysis, and support responsible deployment of GenAI systems in medical education.

Conclusions

In this preliminary, single-center, transcript-based study of a Japanese-language OSCE dataset, uncalibrated single-run zero-shot evaluations by GPT-5.2 Thinking and Gemini 3.0 Pro assigned lower scores than clinical educator consensus ratings. These findings do not support the use of these models as standalone evaluators under the specific conditions tested for Japanese medical interview training. However, the results do not establish that these models generally lack validity across other settings, languages, prompts, model configurations, or assessment formats. Further multicenter studies, refinements in prompting strategies, and model calibration, particularly through specialized training, are required to harmonize GenAI evaluation with human clinical judgment.