This prospective methodological study evaluated the performance of 3 LLMs in generating patient-oriented information about cardiovascular imaging. We followed the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist for observational studies from the EQUATOR (Enhancing the Quality and Transparency of Health Research) network [29] and have provided it in Multimedia Appendix 1.

This study did not involve human participants, the collection or analysis of identifiable private patient information, or any clinical intervention. All inputs were publicly available patient education materials, and all outputs were AI-generated text. The study, therefore, did not meet the regulatory definition of human subject research, and prospective institutional review board or research ethics board approval was not sought. This determination is consistent with the US Department of Health and Human Services Common Rule (Title 45 of the Code of Federal Regulations §46.102(e) [30], under which an activity constitutes human subject research only when an investigator obtains information or biospecimens through interaction with living individuals or obtains identifiable private information, and with the approach taken in prior evaluations of conversational agents in health care communication [25,26]. Because no human participants were enrolled and no identifiable data were used, informed consent was not applicable.

Patient-centered questions were compiled from established cardiovascular information resources, including the American College of Cardiology, the American Heart Association, and MedlinePlus. These sources provide standardized, guideline-aligned educational material for patients with cardiovascular diseases [15-17]. To ensure broad representation of real-world patient concerns, additional questions were gathered from publicly accessible cardiovascular support forums and online patient communities, reflecting common uncertainties expressed in digital health environments.

All collected questions underwent independent review by 3 authors with experience in cardiovascular imaging and patient communication. During this review, questions that were repetitive, ambiguous, subjective, or not directly related to cardiovascular imaging were excluded. The final dataset consisted of 84 unique patient-oriented questions. The use of structured, curated question sets follows established approaches in prior LLM evaluation studies examining accuracy and patient communication quality [18,20,25,31]. Minor edits were applied to improve grammar and readability while preserving original meaning. The included questions encompassed procedural preparation, imaging risks, diagnostic purpose, radiation exposure, and lifestyle considerations associated with cardiovascular imaging.

Each of the 84 questions was submitted to 3 LLMs: GPT-4o, GPT-o1, and DeepSeek. All responses were generated on May 9, 2025. To prevent contextual contamination across queries, each question was entered into a separate, newly initiated chat session, an approach consistent with single-turn evaluation methodologies used in previous radiology-focused LLM studies [8,20,32]. No follow-up prompts, clarifications, or feedback were provided. All questions were submitted in English. Each response was generated once and was not regenerated. Default platform settings were used for all models, and no generation parameters (eg, temperature or sampling settings) were manually modified by the investigators. Figures 1-3 show the different responses to one example question from the 3 models. The raw responses to all questions from each model can be found in Multimedia Appendices 2-4.

To evaluate the quality of AI-generated responses to patient inquiries regarding cardiovascular imaging, a structured assessment framework was developed based on established approaches used in prior analyses of conversational agents, medical question–answering systems, and radiology-focused LLM evaluations [25,26,31,32]. The framework consisted of 4 primary evaluation criteria: accuracy, clarity and appropriateness, completeness, and user engagement and reassurance.

Two independent cardiovascular radiologists conducted the evaluations using this standardized scoring rubric. Using multiple expert reviewers and predefined scoring criteria aligns with recommended methodological practices for assessing communication quality and ensuring reproducibility in AI-mediated health information research [24,33]. Figure 4 illustrates the overall methodological workflow of the study, including question selection, response generation by the 3 LLMs, expert evaluation, and statistical analysis.

To minimize evaluator subjectivity, a predefined discrepancy resolution procedure was implemented. This approach mirrors adjudication methodologies used in diagnostic performance research and multi-rater AI evaluation studies [24,35]. Disagreements were resolved through a structured process. The 2 cardiovascular radiologists first scored each response independently. Where scores differed by more than 2 points on any category, the evaluators discussed their rationale to reach consensus. For all other disagreements, an independent cardiovascular imaging expert served as the final arbitrator.

The composite score was calculated by summing the scores across the 4 evaluation domains (accuracy, clarity and appropriateness, completeness, and user engagement and reassurance), resulting in a total score ranging from 4 to 12. Composite scores were interpreted as follows: a score of 12 indicated excellent performance, reflecting responses that were fully accurate, clear, comprehensive, and patient centered; scores of 10 to 11 indicated good performance with only minor limitations; scores of 7 to 9 indicated adequate performance requiring some improvement; and scores of 4 to 6 indicated poor performance with substantial deficiencies across one or more evaluation domains.

For each response and domain, the value entered into the analysis was the single adjudicated score: in cases in which the 2 primary reviewers agreed, that score was used directly, and in cases in which they disagreed, the score assigned by the senior reviewer (R3) was used. The composite total score for each response was then obtained by summing the 4 adjudicated domain scores (each scored from 1-3), yielding a possible range of 4 to 12. Because the domains were measured on a 3-point ordinal scale, scores for each domain and for the composite total were summarized as medians with IQRs rather than as means with SDs. Differences in performance across the 3 LLMs were assessed using the Kruskal-Wallis test, a nonparametric method appropriate for comparing ordinal scores across 3 independent groups. Interrater reliability was assessed using the Cohen κ with quadratic weighting, which accounts for the ordinal structure of the 3-point scoring rubric and penalizes larger discrepancies more heavily than minor disagreements. Agreement was quantified between each primary reviewer (R1 and R2) and the adjudicating senior cardiovascular radiologist (R3). Agreement between R1 and R2 was not evaluated as the primary objective of the reliability analysis was to assess consistency relative to the final adjudicated reference standard rather than raw concordance between initial reviewers. The adjudicating radiologist (R3), as the most experienced reviewer, resolved all discrepancies and generated the final scores used for downstream analysis; therefore, agreement with R3 was considered the most clinically and methodologically relevant measure of scoring reliability. For each domain, and for the composite score, the Kruskal-Wallis H statistic, its df, the associated P value, and ε² as an effect size estimate are reported. The distribution of ordinal ratings across score categories is also presented as counts and percentages for descriptive purposes only to show how responses were spread across the rating categories within each domain; these percentages were not subjected to separate inferential testing. All statistical analyses were performed using R (version 4.5.2; R Foundation for Statistical Computing), with statistical significance defined as an α value of .05.

The evaluation sheets completed independently by the 2 primary cardiovascular radiologists, with adjudication by a third radiologist, can be found in Multimedia Appendices 5-10. Evaluations for all 3 models by the remaining reviewer are provided as separate worksheets in Multimedia Appendix 11.

Interrater reliability analysis (Table 2) demonstrated moderate agreement across all models based on commonly used interpretive benchmarks for the Cohen κ (eg, values of 0.41-0.60 indicating moderate agreement and 0.61-0.80 indicating substantial agreement). Notably, GPT-4o exhibited slightly higher quadratic weighted Cohen κ values than DeepSeek and GPT-o1. This finding likely reflects the relative uniformity and neutrality of GPT-4o’s responses, which may be easier to score consistently, rather than superior communication quality. In contrast, the more expressive and patient-centered responses generated by DeepSeek and GPT-o1 may introduce greater subjective variability between raters despite higher overall performance. Table 3 presents the descriptive statistics for all 4 scoring domains and the total composite score for each model. These differences were modest and should be interpreted cautiously as interrater reliability reflects consistency of scoring rather than intrinsic quality, clinical accuracy, or educational effectiveness of the model responses. Importantly, the observed agreement levels across all models indicate that the scoring rubric could be applied reproducibly by independent reviewers, supporting the robustness of the evaluation framework rather than implying clinically meaningful distinctions between models.

GPT-o1 demonstrated a consistently strong performance across all domains. Its responses were frequently rated as fully accurate, comprehensive, and clearly structured. Notably, it achieved the highest level of user engagement and reassurance among the 3 models, indicating a communication style that was patient-friendly and supportive. Overall, GPT-o1 produced balanced, high-quality outputs with strengths in clarity and patient-centered phrasing. DeepSeek showed a similarly favorable performance profile, with particularly strong clarity and readability. The model produced well-organized explanations that were easy for lay users to understand. Accuracy and completeness were comparable to those of GPT-o1, and its user engagement scores were also high, although marginally lower than those of GPT-o1. DeepSeek’s overall performance indicates a reliable ability to deliver clear and reassuring patient education content. GPT-4o demonstrated accuracy and completeness comparable to those of the other 2 models but differed meaningfully in its communication tone. While technically sound, its responses were more neutral and less patient oriented, resulting in lower scores in the user engagement and reassurance domain. This reduced patient-centeredness contributed to a lower total score relative to GPT-o1 and DeepSeek despite similar technical quality.

Overall, the descriptive statistics in Table 3 show that all 3 models are capable of producing accurate and complete responses. However, GPT-o1 and DeepSeek received higher ratings than GPT-4o in affective and engagement-related qualities, which may be particularly important in patient-facing educational contexts.

Table 4 presents the categorical distribution of scores across the 3 LLMs for descriptive purposes. For the accuracy domain, inaccurate responses were rare and limited to GPT-4o, whereas the proportion of fully accurate responses was similarly high across all 3 models.

For clarity and appropriateness, most responses from all models were rated as clear and concise, and unclear or confusing responses were rare across all 3 models. For completeness, most responses from every model were rated as comprehensive, with the remainder rated as partially comprehensive. The only domain with a meaningful difference was user engagement and reassurance. DeepSeek and GPT-o1 generated predominantly high-engagement responses, whereas GPT-4o produced a substantially lower proportion of high-engagement responses and a correspondingly higher proportion of moderate ones. This communication-related deficit contributed directly to the lower overall ratings of GPT-4o. Differences were also reflected in total score distributions. DeepSeek and GPT-o1 exhibited the highest proportion of perfect scores, whereas GPT-4o achieved a perfect score less often and showed a higher frequency of midrange scores, indicating that its responses, while technically sound, were less consistently reassuring and supportive for patients.

Figure 5 illustrates the distribution of total composite scores across models. While DeepSeek and GPT-o1 showed highly overlapping score distributions and did not differ significantly from one another, both models achieved higher total scores than GPT-4o. The broader spread and downward shift observed for GPT-4o align with its lower engagement and reassurance ratings, indicating greater variability and less consistently patient-centered communication.

This study evaluated the quality of patient-facing cardiovascular imaging information generated by GPT-o1, GPT-4o, and DeepSeek across the domains of accuracy, clarity, completeness, and user engagement. Overall, all 3 models demonstrated similarly strong technical performance, with no statistically significant differences in accuracy, clarity, or completeness, indicating that contemporary LLMs are consistently capable of providing factually correct, clear, and comprehensive explanations of cardiovascular imaging for patients. The primary distinction between the models emerged in the user engagement and reassurance domain, where DeepSeek and GPT-o1 consistently generated more supportive, empathetic, and patient-centered responses than GPT-4o. Although the absolute differences in composite scores were modest, they were driven largely by these communication-related characteristics rather than by differences in technical correctness. These findings suggest that state-of-the-art LLMs have largely converged in their ability to deliver accurate educational content and that communication style, reassurance, and patient-centered language may now represent the principal factors differentiating their suitability for patient education. This distinction is particularly relevant in cardiovascular imaging, where patients often seek explanations of unfamiliar procedures while experiencing uncertainty or anxiety, making both the quality of information and the manner in which it is communicated important components of effective patient education.

The consistently strong performance observed across the accuracy, clarity, and completeness domains aligns with a growing body of literature demonstrating that modern LLMs can effectively generate medically appropriate educational content for patients. Prior evaluations across multiple medical specialties have reported that leading LLMs frequently provide responses that are accurate, comprehensive, and accessible to nonexpert audiences, supporting their potential utility as patient education resources [24,33]. Similar observations have been reported in studies evaluating patient-facing questions in radiology, oncology, cardiology, and surgical care, where LLM-generated responses often achieved high ratings for factual correctness and readability while maintaining language that could be understood by lay users [24,33].

The absence of meaningful differences between models in these technical domains suggests that all 3 systems were capable of conveying essential information regarding cardiovascular imaging procedures. This finding is encouraging because patient education represents one of the most promising and comparatively lower-risk applications of LLM technology within health care. Unlike diagnostic decision-making, patient education primarily involves translating established clinical knowledge into understandable explanations. The ability of multiple independent models to perform similarly in this setting may therefore reflect increasing maturity of LLMs for educational applications.

Although technical performance was largely comparable across models, differences emerged in user engagement and reassurance. Responses perceived as more supportive, empathetic, and patient centered generally received higher overall evaluations despite containing information that was often similar in factual content. This observation is consistent with previous research indicating that patients value communication qualities such as empathy, emotional support, reassurance, and conversational tone in addition to factual accuracy [24,33].

Importantly, effective patient education extends beyond the transmission of correct information. Patients undergoing cardiovascular imaging procedures may experience anxiety related to diagnostic uncertainty, radiation exposure, contrast administration, procedural discomfort, or potential findings. In these contexts, responses that acknowledge concerns and provide reassurance may contribute meaningfully to patient understanding and satisfaction. Prior studies comparing LLM-generated and clinician-generated responses have similarly reported that communication style and perceived empathy often influence overall evaluations of response quality even when factual content is comparable [24,33].

These findings suggest that evaluation frameworks for patient-facing AI systems should not focus exclusively on factual correctness. Measures of engagement, reassurance, readability, and patient-centered communication may be equally important when assessing the real-world utility of LLM-generated educational content. Future model development efforts may therefore benefit from explicitly optimizing both informational quality and supportive communication.

Taken together, the findings support the potential role of LLMs as adjunct tools for cardiovascular imaging education. Such systems may help patients obtain preliminary information, reinforce explanations provided by health care professionals, and improve access to educational resources outside clinical encounters. However, their greatest value is likely to arise when they complement rather than replace clinician-patient communication.

The results further suggest that future evaluations of patient-facing LLMs should incorporate broader measures of communication effectiveness, including patient comprehension, trust, anxiety reduction, and perceived usefulness. Although technical accuracy remains essential, educational success ultimately depends on whether information is communicated in a manner that patients find understandable, reassuring, and actionable.

Although no hallucinations or clinically concerning inaccuracies were identified in the responses evaluated in this study, hallucinations remain a recognized limitation of LLMs in health care and have been documented even in high-performing systems evaluated across clinical and biomedical domains [23,34,35]. Consequently, patient-facing deployment should incorporate appropriate safeguards to minimize the risk of incorrect or unsupported information. Proposed mitigation strategies include retrieval-augmented generation approaches that ground responses in curated evidence sources, structured prompting techniques, citation-aware generation, and ongoing human oversight [36-38]. Continued monitoring, domain-specific refinement, and transparent reporting of safety limitations will remain important as LLMs become increasingly integrated into health care communication workflows [39,40].

The ability of LLMs to provide accessible educational information may have particular relevance in settings where access to health care professionals and patient education resources is limited. Prior work has highlighted the potential value of AI-based tools in health care systems facing shortages of trained specialists and educational resources [41]. However, the present study did not directly evaluate model performance across different languages, cultures, health care systems, or resource-constrained environments. Differences in digital literacy, internet access, and linguistic availability may substantially influence real-world utility. Consequently, further work is needed before conclusions regarding the applicability of these findings to low- and middle-income countries can be drawn.

This study has several limitations. First, LLM behavior evolves rapidly as models undergo continuous updates and retraining. These findings represent a snapshot of model behavior at a specific time point and should not be interpreted as immutable performance characteristics. Second, the analysis was limited to English-language questions, which may not generalize to multilingual or culturally adapted settings. Third, while this study evaluated patient-oriented communication, it did not assess clinical reasoning depth or medical decision-making accuracy, which are distinct dimensions of LLM performance. Fourth, even with a structured rubric, the scoring depended on expert judgment, which introduces subjectivity and may not capture all subtleties of communication quality. In particular, domains such as user engagement and reassurance represent inherently subjective constructs, and their evaluation reflects a clinical communication perspective rather than direct end user perception. Patient-based scoring or feedback was not incorporated in the present study, and future work should examine how patients themselves perceive engagement, reassurance, and the overall usefulness of LLM-generated explanations. Finally, despite discussing the potential relevance for low- and middle-income country settings, the present study did not directly evaluate model utility, accessibility, or safety in these environments, and thus, conclusions about their broader global applicability should be interpreted with caution.

These findings carry several broader implications for the use of LLMs in patient education. As the models differed primarily in the affective and patient-centered quality of their communication rather than in clinical correctness, communication tone, empathy, and reassurance should be considered explicit design and evaluation targets for patient-facing AI systems. Realizing the potential of these tools at scale, particularly in settings where access to specialist education is limited, will require safeguards against hallucinations, strategies that promote supportive and comprehensible responses, and adaptation to diverse health literacy levels and languages. Prospective studies involving patients and evaluating real-world outcomes such as comprehension, anxiety reduction, and decision-making will be essential before these technologies can be safely integrated into routine cardiovascular imaging education.