Principal Findings
In this study, we evaluated the performance of ChatGPT, Claude, and AMBOSS on 200 EBU-style MCQs using a multidimensional assessment framework that examined not only answer accuracy but also concordance, nonobvious deduction, discriminative reasoning, and clinical validity. Although previous studies have demonstrated that LLMs can achieve pass-level or near–pass-level performance on medical and specialty examinations, these evaluations have largely focused on answer correctness and examination outcomes [20-23]. In contrast, our study assessed the quality and transparency of the reasoning underpinning AI-generated responses, with a particular emphasis on adherence to EAU guideline–based practice.
Previous work supports the growing role of AI in medical education, with GPT-based models demonstrating strong performance across licensing and board examinations [20-23].
However, most studies have focused primarily on examination accuracy rather than the reasoning processes underlying responses. For example, Vaishya et al [24] evaluated AI performance on orthopedic postgraduate examination questions without systematically assessing explanation quality. Similarly, studies of medical question-answering systems and patient-facing chatbots have shown encouraging performance but have also highlighted that answer correctness alone may not adequately capture response quality, transparency, safety, or clinical applicability [25,26].
Assessing guideline adherence is particularly important given the known limitations of AI-generated clinical recommendations. Talyshinskii et al [27] found that although GPT-4.0 demonstrated partially accurate diagnostic knowledge in urolithiasis, its surgical planning recommendations were not consistently aligned with EAU guidelines. Additionally, Eldaneen et al [28] reported that current LLMs provide readily accessible guidance on LUTS; however, their unsupervised use in clinical decision-making remains a concern and may be considered premature.
Broader evaluations of generative AI in health care have similarly highlighted concerns regarding reliability, hallucinations, safety, and the need for human oversight [29,30]. Furthermore, Topol [31] argued that successful integration of AI into health care depends on maintaining human clinical oversight, while systematic reviews have identified ongoing concerns regarding reliability, interpretability, and real-world applicability [32]. Together, these findings emphasize the importance of evaluating not only whether an answer is correct but also whether the underlying reasoning is transparent and clinically appropriate.
Although all 3 models achieved an accuracy exceeding 70%, important differences emerged when reasoning quality was assessed directly. ChatGPT demonstrated the strongest overall performance, with the highest accuracy, concordance, insight scores, and clinical validity. This finding is consistent with recent reports showing strong performance of GPT-4.0–based models across medical knowledge and clinical reasoning assessments [33]. Although all models achieved scores exceeding 70% on the EBU In-Service Assessment workbook, these results should not be interpreted as equivalent to performance on the official EBU part 1 examination because the workbook is an educational resource rather than a validated examination used for certification.
ChatGPT more frequently incorporated structured, multistep reasoning; guideline-based interpretation; and explicit exclusion of alternative answers, resulting in the highest concordance scores.
Claude achieved comparable accuracy and demonstrated strong clinical validity and discriminative reasoning. Its explanations frequently justified why alternative answers were incorrect and were generally consistent with accepted urological practice. However, compared with ChatGPT, Claude was less likely to generate reasoning that extended beyond the information explicitly provided in the question stem.
AMBOSS also demonstrated good overall accuracy and guideline alignment, but its explanations were typically shorter and more descriptive. Although correct answers were often provided, the explanations were less likely to justify the selected option or actively exclude competing alternatives. This was reflected in lower concordance and insight scores than those of ChatGPT and Claude.
High levels of nonobvious deduction were uncommon across all models. This likely reflects the structure of EBU-style questions, many of which assess factual recall and therefore provide limited opportunities for advanced reasoning. Higher ratings were generally observed in questions requiring the integration of multiple clinical concepts or the application of guideline-based management decisions.
The greatest differences between models were observed in discriminative reasoning. ChatGPT and Claude more frequently compared competing answer options and explicitly explained why incorrect alternatives should be excluded. In contrast, AMBOSS often identified the correct answer without fully articulating the reasoning process that led to that conclusion. The ability to actively exclude alternative options represents an important component of both examination performance and clinical decision-making and may enhance the educational value of AI-generated explanations.
All 3 models demonstrated strong clinical validity overall, with most responses aligning with accepted urological practice and EAU guideline recommendations. ChatGPT demonstrated the highest consistency in this domain. These findings are reassuring from an educational perspective, suggesting that clinically inappropriate or potentially misleading recommendations were relatively uncommon across all 3 platforms.
Our findings provide further insight into previously reported limitations of AI performance in medical assessments. Although studies such as those by Sadeq et al [34] have demonstrated variability in chatbot performance across question types and levels of complexity, our results suggest that differences between models are not fully explained by accuracy alone. Models with similar examination performance may differ substantially in how they justify their answers, exclude alternative options, and maintain alignment with guideline-based practice. This observation is consistent with the work of Holzinger et al [35], who emphasized the importance of evaluating explanations rather than outputs alone.
On the basis of the findings of this study, AMBOSS may be particularly useful for factual revision, whereas ChatGPT and Claude may offer additional value for examination preparation because of their more detailed reasoning processes. Nevertheless, none of the evaluated models should be considered substitutes for clinical supervision or contemporary guideline consultation.
Overall, AI should be introduced in a staged and supervised manner, supporting rather than replacing traditional educational approaches, consistent with recent recommendations regarding the integration of LLMs into medical education [36,37].