Comparing ChatGPT and DeepSeek for Assessment of Multiple-Choice Questions in Orthopedic Medical Education: Cross-Sectional Study

Introduction

Multiple-choice questions (MCQs) are a cornerstone of medical education, particularly for assessing knowledge acquisition and clinical reasoning in specialized fields such as orthopedic medicine. Traditionally, MCQs have been refined through expert reviews and iterative revisions, a process that can be time-consuming and prone to human bias [1,2]. MCQs are widely used in medical education because of their efficiency, objectivity, and ability to assess a broad range of knowledge in a standardized manner [2,3].

Large language models (LLMs), such as ChatGPT and DeepSeek, have emerged as potential tools for evaluating MCQ accuracy and efficiency. These artificial intelligence (AI)–driven approaches can systematically assess question clarity, difficulty, and alignment with learning objectives [4]. However, research comparing the performance of different LLMs in answering orthopedic MCQs remains limited.

Recent studies have highlighted the capabilities of LLMs in medical education [5-7]. These studies explored the use of GPT-4 for formative and summative assessments in medicine, demonstrating its potential in evaluating medical knowledge and providing feedback to students. Additionally, ChatGPT has shown notable accuracy in medical education, particularly in answering MCQs for licensing exams [8]. Prior studies have shown its ability to perform well in both basic and clinical sciences, with accuracy rates ranging from 70% to 74% [9]. Despite its potential, the reliability of LLMs in MCQ assessment, particularly in disciplines requiring complex clinical reasoning, remains an area of ongoing investigation [8,9].

DeepSeek V3, a recent advancement in LLMs, has shown promise in various domains in health care. Research has explored its use in the medical context, including patient education on asthma, ocular oncology, and laparoscopic cholecystectomy [10-12]. However, no study has evaluated the performance of DeepSeek V3 as an assessment tool for the quality of MCQs in medical education.

This study compared the performance of ChatGPT and DeepSeek in answering orthopedic MCQs by evaluating their correctness, response time, and reliability. These findings provide insights into the feasibility of integrating LLMs into medical student assessments and highlight areas where MCQs may require refinement.

Methods

Study Design

This cross-sectional study was conducted at the Department of Orthopedics, Faculty of Medicine, Prince of Songkla University, Thailand. It focused on MCQs used in the summative assessment of orthopedic rotations during the 2023‐2024 academic year. All MCQs from the question pool during this period were included. As this was an educational study, there were no criteria for participant withdrawal or study termination. Instead, all available MCQs from the study period were analyzed.

Assessment Tools

In this study, MCQs were defined as structured assessment tools consisting of a question stem and multiple answer choices with one correct response. This study evaluated the ability of two LLMs, ChatGPT and DeepSeek, to analyze and process these questions. The versions used were GPT-4o for ChatGPT and and DeepSeek-V3 for DeepSeek, as both represented the most stable and publicly accessible variants during the data collection period. The two LLMs were accessed between January 2025 and March 2025.

GPT-4o was developed by OpenAI and released in May 2024 [13]. This model was optimized for faster response times and improved performance while maintaining high accuracy in complex reasoning tasks. Additionally, ChatGPT includes a “Reason” function that allows users to request explanations for its responses, providing insights into the logic behind its choices. This feature enhances transparency and facilitates a deeper understanding of how the model arrived at its conclusions, making it particularly useful for educational and assessment purposes [14].

DeepSeek-V3, another advanced LLM used in medicine, was initially released in December 2024 [13]. It offers enhanced performance with improved accuracy, faster processing, and a broad, up-to-date knowledge base [15]. It includes a “DeepThink” function designed to enhance logical reasoning by breaking down complex problems step by step. The AI model is designed to deliver high-quality, reliable, and efficient assistance in various applications. It excels in natural language processing, providing superior contextual understanding and multilingual support, ensuring more natural and coherent interactions.

Data Collection and Evaluation Procedures

This study involved 3 main steps. First, a set of MCQs from orthopedic assessments conducted in 2023‐2024 was collected and reviewed for quality. These MCQs were created by orthopedic faculty members and approved by the orthopedic committee to ensure content alignment with curricular objectives, and they have been used in orthopedic examinations for more than 5 years. Next, each MCQ was input into both ChatGPT and DeepSeek. The AI responses, along with the accuracy and consistency data, were recorded. The Reason and DeepThink functions were tested, and response times were documented. Finally, any MCQs for which both LLMs failed to identify the correct answer were reviewed by a panel of orthopedic faculty members. These questions were refined to improve clarity and adjusted to ensure an appropriate level of difficulty. To determine reliability, all the MCQs and their answer keys were input into both ChatGPT and DeepSeek by two orthopedic faculty members, who both had experience in medical education at an academic hospital, on separate days.

Data Analysis

Statistical methods were used to compare the performance of the two LLMs. Correctness rates were expressed as percentages and compared using the χ² test. Processing times were reported as means with SDs and analyzed using either a 2-sample, 2-tailed t test or a Mann-Whitney U test, depending on data distribution. The interrater reliability between the two LLMs was evaluated using the Cohen κ coefficient. Statistical significance was set at P<.05.

Ethical Considerations

This study was conducted in accordance with the ethical principles of the Declaration of Helsinki, and the protocol was reviewed and approved by the Institutional Review Board of the Faculty of Medicine at Prince of Songkla University (REC. 68-097-11-1). This study involved only secondary analysis of existing MCQs and contained no participant identifiers; thus, individual informed consent was waived. As this study did not involve direct participation by human subjects, no compensation was provided. Finally, no images or materials that could identify individual participants were included in the manuscript or supplementary files.

Results

A total of 209 orthopedic MCQs were analyzed across 11 subspecialty categories to compare the accuracy of ChatGPT and DeepSeek (Table 1) and the Reason and DeepThink functions (Table 2). Overall, ChatGPT achieved a significantly higher total accuracy of 80.4% (168/209) compared with 74.2% (155/209) for DeepSeek (P=.04). Although no single subspecialty demonstrated a significant difference after stratification, ChatGPT showed numerically superior accuracy across most categories, with the largest gaps observed in the tumor and infection (92.9% vs 71.4%) and lower-limb injury (68.8% vs 50.0%) domains. These findings indicate that ChatGPT outperformed DeepSeek in overall accuracy for orthopedic MCQs.

In the second comparison, the Reason function in ChatGPT demonstrated an accuracy of 84.7% (177/209), while the DeepThink function in DeepSeek achieved 80.4% (168/209) accuracy. However, the difference was not statistically significant (P=.12). Nevertheless, the Reason function seems to have performed better in some domains, such as the pediatric and anomalous category (80.0% vs 65.0%) and the tumor and infection category (92.9% vs 78.6%), whereas DeepThink showed slightly higher accuracy in the complications in orthopedics category (90.9% vs 63.6%). However, these discrepancies did not appear to correlate with any specific subspecialty or content category.

The most incorrect responses for both ChatGPT and DeepSeek occurred in the complications (2/11, 18.2%), pediatric and anomalous (3/20, 15.0%), and pelvic and spine injury (3/19, 15.8%) categories, reflecting areas where clinical reasoning and management knowledge were more heavily tested. The two lowest rates of mistakes were observed in the tumor and infection and upper limb injury categories. The overall rate of completely false responses from the two LLMs was 7.7% (16/209). Examples of questions that resulted in false responses from both LLMs are provided in Multimedia Appendix 1.

ChatGPT demonstrated a significantly faster response time, with an average processing time of 10.40 (SD 13.29) seconds, compared to DeepSeek’s 34.42 (SD 25.48) seconds (P<.001).

Finally, the interrater reliability for the two LLMs was high. ChatGPT achieved a Cohen κ value of 0.81, indicating good agreement, whereas DeepSeek showed substantial agreement, with a κ of 0.78. These findings underscore both the consistency of the LLM-generated responses and a residual variability that warrants continued human oversight in educational and assessment settings.

Discussion

Principal Results

This study compared the performance of two advanced LLMs, ChatGPT and DeepSeek, in the processing of orthopedic MCQs regarding their correctness, response time, and reliability. Our findings indicate that ChatGPT outperformed DeepSeek in terms of both accuracy and response time. Additionally, the Reason function of ChatGPT, which provides explanations for selected answers, appears to contribute to its superior performance. Both ChatGPT and DeepSeek exhibited high interrater reliability when generating responses to orthopedic MCQs.

Comparison With Previous Studies

The results of this study are consistent with a prior study showing that advanced LLMs demonstrated a high level of accuracy in answering orthopedics-related MCQs [16]. In our study, the accuracy of ChatGPT without the Reason function was 80.4%, while a previous study reported an accuracy rate of 70.60%. When using the Reason function in ChatGPT, the accuracy increased to 84.7%. This may imply that the Reason function can promote deeper comprehension and facilitate critical thinking in the medical education setting.

The high accuracy observed for ChatGPT and DeepSeek is particularly valuable in high-stakes environments like medical education [17]. Efficient processing can facilitate the development of high-quality, comprehensive MCQs, minimizing ambiguities in both questions and answer choices to enhance assessment reliability. Furthermore, the ability of ChatGPT’s Reason function to provide detailed explanations adds an educational dimension often missing in traditional assessment tools, thereby facilitating a deeper understanding of complex clinical contexts that mirror real-world practice [18].

The balance between AI efficiency and educational values poses an issue. Although ChatGPT’s speed and accuracy are advantageous, reliance on automated systems for assessment must be balanced with human oversight to ensure that the nuances of clinical reasoning and decision-making are adequately taken into account. This is particularly important in fields such as orthopedic surgery, where clinical decisions can have significant implications for patient care [19].

We found that 7.7% of the MCQs yielded false responses from both LLMs, warranting further examination. This finding suggests that there may be inherent issues in question construction, leading to ambiguity and misinterpretation. Poorly constructed questions can stem from several factors, including unclear wording, imprecise phrasing of the question stem, or answer choices that do not effectively discriminate between correct and incorrect responses [20,21]. By revising ambiguous or poorly constructed questions, educators can improve the overall reliability and validity of assessments, subsequently enhancing the fairness of the evaluation process [22].

Both ChatGPT and DeepSeek exhibited high interrater reliability when generating responses to orthopedic MCQs. These results suggest that both AI models can consistently answer the standardized questions, although some variability remains. The differences in the κ values may have resulted from variations in the training data and model algorithms. Despite their strong reliability, the models did not achieve complete consistency, highlighting the importance of human oversight.

Although AI tools provide notable benefits in terms of efficiency and consistency, there are critical risks associated with overreliance on them for medical education and assessment. Therefore, human supervision is necessary to guarantee that AI functions as a helpful tool rather than a substitute for professional judgment, particularly in situations involving high stakes in education or health care. Educators should carefully combine AI with conventional teaching techniques and evaluation procedures to maintain academic integrity and guarantee that students keep refining their decision-making abilities.

Limitations

Despite the promising findings, this study had some limitations. Our study focused solely on orthopedic MCQs, which may limit the generalizability of our results to other areas of medical education. Future studies should extend these analyses to include diverse medical disciplines to better understand the broader applicability of AI-assisted assessments. Additional limitations include the study’s cross-sectional design, which only captured the LLMs’ performance at a single point in time. Given that LLMs such as ChatGPT and DeepSeek are continually updated and improved, their performance may vary over time. Furthermore, while the LLMs provided explanations for selected answers, we did not formally assess the quality of these explanations. Implementing a structured evaluation of explanation quality in future studies could offer valuable educational perspectives.

Implications of the Findings

The findings suggest that AI tools like ChatGPT and DeepSeek can enhance the efficiency of medical assessments in orthopedic rotation. ChatGPT demonstrated higher accuracy and faster responses compared to DeepSeek. Furthermore, both LLMs could help identify poorly constructed MCQs. However, reevaluation by teachers or assessors remains essential to ensure that responses are clinically relevant and that results of summative assessments are accurate.

For broader application in other medical specialties, we propose the following practical steps. First, validated, specialty-specific MCQs should be gathered from the target discipline to ensure content relevance and quality. Second, these questions should be input into selected AI models and their responses systematically recorded and evaluated for accuracy. Third, to ensure reliability, at least two faculty experts should independently assess the LLM-generated answers, with any discrepancies resolved through consensus review. Fourth, poorly performing or ambiguous MCQs should be identified and refined for clarity.

Currently, there are a large number of advanced LLMs. Future studies should systematically compare other AI tools to determine which models perform best in MCQ assessment. Additionally, future research should be conducted to explore whether AI assistance could be used for other types of assessments, including modified essay questions, objective structured clinical examinations, or key-feature questions.

Conclusions

In the study, ChatGPT outperformed DeepSeek in answering orthopedic MCQs in terms of overall accuracy and response time, although both models exhibited strong interrater reliability. Specifically, ChatGPT achieved a significantly higher accuracy rate compared with DeepSeek, and its Reason function further enhanced performance. These findings suggest that ChatGPT can serve as a valuable tool in medical education and assessment, particularly when using explanatory reasoning capabilities.

Despite these encouraging results, both AI models showed areas of weakness, particularly in complex domains such as complications, pediatrics and anomalies, and pelvic and spine injuries. These errors highlight the importance of maintaining human oversight when using LLMs for educational or evaluative purposes. Thus, the integration of AI tools should be viewed as complementary to, rather than a replacement for, expert clinical judgment and expertise.