An evaluation study was conducted to evaluate the interpretability and readability of information leaflets on PICS generated using LLMs. Using prompt engineering and text-augmented prompting approach, we generated patient-friendly explanations of clinical practice guidelines and evaluated the quality of the generated materials.

Four prompts were designed to explore different content generation strategies and levels of guidance (Multimedia Appendix 1).

These prompts were applied to 6 LLMs—ChatGPT-4o, LLaMA 3 70B, MedItron:7b, Gemma, Mistral, and MedLLaMA2—with 3 outputs generated per prompt-model combination, resulting in a total of 72 texts (Multimedia Appendix 2). To reflect the models most likely to be used by health care professionals, we focused on available models through Ollama and selected the 4 models with the highest number of pulls as of May 2024, as well as the 2 models with the highest number of pulls identified by searching for the keyword “medical.” For open-source models, we used Ollama with its default generation settings at the time of execution (July 2024) without parameter tuning. The settings included a temperature of 0.8, repeat penalty of 1.1, repeat_last_n of 64, seed of 0, and a context window size (num_ctx) of 2048 tokens. We did not perform token counting or explicit verification of context retention during generation. Therefore, depending on the combined length of the system instructions, prompts, uploaded guideline text, and generated output, partial truncation of the contextual input may have occurred. This configuration was maintained to simulate a standard user environment where context window adjustments are not typically performed.

For prompt conditions using guideline context (prompts 2‐4), the generation workflow was as follows:

No embedding-based retrieval, vector database indexing, or automated selection of guideline passages was performed. Parameter tuning and advanced retrieval pipelines were not intentionally implemented to reflect a pragmatic clinical scenario in which health care professionals use commercially available LLMs with minimal technical customization [16].

For human evaluation, we compared the relative quality of practically usable outputs across model-prompt configurations. Because human evaluation requires substantial time and effort, the number of texts was restricted to a feasible sample size, and clearly incomplete outputs were excluded prior to evaluation.

The generated texts were selected through a structured 3-step process. First, 72 texts (6 models × 4 prompts × 3 outputs, Multimedia Appendix 2) were screened for minimum length (≥500 Japanese characters) and explicit mention of PICS, leaving 53 texts. Second, for each model-prompt combination, the longest output was selected as a proxy for informational coverage to reduce the likelihood of including incomplete or truncated responses and to ensure sufficient content for human evaluation, resulting in 21 texts. Third, to ensure balanced representation across prompts and models, four models (GPT-4, Llama3:7b, MedLLaMA, and Mistral) were retained. Because all MedLLaMA outputs for prompt 3 had been excluded during initial screening, the longest excluded output was reinstated to maintain balance. This process yielded 16 final texts (4 models × 4 prompts, Multimedia Appendix 3). These texts were initially generated in English and subsequently formatted into Japanese A4-sized informational leaflets using a GPT-4o–based system. The translation quality of GPT-4 has been evaluated in prior research against human translators and was found to be comparable to junior professional translators in terms of overall error rates, supporting the reliability of this approach [17]. Subsequently, 2 board-certified emergency physicians (NH and TO) independently reviewed the 16 leaflets for accuracy, clarity, and completeness. Any disagreements between the reviewers were resolved through discussion to reach a consensus. Nine leaflets were selected for further evaluation (Figure 1, Table 1, and Multimedia Appendix 4). A few were excluded because of limited informational content or language that was more suited for medical professionals; however, there were no critical errors that could potentially cause harm to patients or their families. Details of the screening process and the rationale for expert-based selection are provided in Multimedia Appendix 5.

Ten individuals were recruited as evaluators: 2 nonmedical individuals, 2 health care providers (1 nurse and 1 physical therapist), 4 specialists with expertise in PICS (3 emergency physicians and 1 physical therapist), and 2 nonspecialist physicians without PICS-specific expertise. These evaluators were requested to assess 9 selected texts in terms of clarity, readability, and other factors, using 10 evaluation criteria. The evaluation framework was developed with reference to prior large-scale LLM assessment studies in medicine, including the approach described by Singhal et al [4].

The survey was conducted using Google Forms and included the following components.

Because higher raw scores in degree of harmfulness, frequency of harmfulness, inappropriateness, omissions, and potential for bias represented greater risk or lower quality, these items were reverse-scored prior to analysis (transformed as 11—original score) so that higher scores consistently indicated better performance across all domains.

In addition to human evaluation, LLM-based assessments were conducted using GPT-4o for structured comparison with human ratings. Two role-based prompts were used: “You are a medical doctor with extensive experience in emergency care” (physician-role) and “You are a family member of a patient in the intensive care unit” (patient-role).

The 9 selected texts were assessed using the following 6 aggregated dimensions:

Among these, harm-related dimensions, presence of inappropriate or incorrect or missing content, and possibility of bias were reverse-scored (transformed as 11—original score) so that higher values consistently indicated better performance.

Objective assessments of the LLM-generated texts were conducted using human evaluations across 10 predefined criteria and AI-based evaluations incorporating 6 components. Subgroup analyses were performed to compare the evaluation metrics according to the prompt type, LLM used, and the use of text-augmented prompting approach. For graphical presentation of the evaluation results, 95% CIs were estimated using cluster bootstrapping with resampling at the evaluator level (5000 iterations). Additionally, we examined the differences in the evaluation scores according to age, sex, profession, and area of expertise.

As an exploratory analysis, all 72 generated texts (6 models × 4 prompts × 3 outputs) were additionally evaluated using the same GPT-4o–based assessment framework. This analysis aimed to examine evaluation patterns across multiple outputs within each model-prompt pair. To account for clustering of outputs within model-prompt pairs, a mixed-effects model was applied with model-prompt combinations specified as random intercepts. In addition, the mean scores of the 3 outputs within each model-prompt pair were calculated and analyzed using the same framework. Given the limited concordance observed between human and LLM-based evaluations, these analyses were considered exploratory and descriptive.

To assess the interevaluator variability between the human and LLM evaluators, we calculated the weighted κ across the 3 groups (human evaluators, GPT-generated responses simulating a medical doctor, and GPT-generated responses simulating a family member). Interrater reliability among the 10 human evaluators was assessed using an intraclass correlation coefficient (ICC). A 2-way mixed-effects model with absolute agreement (ICC [A,10]) was applied to evaluate consistency across the 10 evaluation criteria.

Categorical variables are reported as absolute numbers and percentages, whereas continuous variables are expressed as mean (SD) or median (interquartile range), as appropriate. All statistical analyses were performed using GraphPad Prism 10 (GraphPad Software), pandas (v1.0.5), numpy (v1.21.4), seaborn (v0.11.2), and matplotlib (v3.5.1) tools in Python (v3.9.0).

This study involved human evaluators who assessed text materials generated by LLMs; however, the evaluators were not the research participants of the investigation. The study focused on the evaluation of AI-generated explanatory texts rather than on human participants themselves. No personal or identifiable information was collected from the evaluators. According to the Ethical Guidelines for Medical and Health Research Involving Human Subjects issued by the Ministry of Education, Culture, Sports, Science and Technology; the Ministry of Health, Labour and Welfare; and the Ministry of Economy, Trade and Industry of Japan (2017, amended 2021) [18], this type of research is considered outside the scope of studies requiring institutional review board approval because it does not involve interventions, collection of personal data, or research targeting human participants.

Therefore, institutional review board approval and formal informed consent were not required.

Ten individuals evaluated the generated texts. Among them, 5 (50%) specialized in emergency and critical care medicine, 1 (10%) in internal medicine, 1 (10%) in psychiatry, and 3 (30%) in other fields. Two (20%) evaluators were involved in the development of the clinical practice guidelines for PICS (Table 2).

All selected texts received mean scores of 6.8 or higher across all evaluation items. Among the 9 outputs, the text generated by LLaMA 3 70B using a step-by-step approach combined with the uploaded context (prompt 4) had the highest overall mean score in the human evaluations. In contrast, the text produced using a simple zero-shot prompt and without the uploaded context (prompt 1) had the lowest overall mean score. These comparisons are descriptive and should be interpreted with caution given the limited number of texts per prompt condition. Items related to readability, difficulty in reading, and omission tended to receive comparatively lower ratings than the other evaluation domains (Figure 2).

The interrater reliability among the 10 human evaluators was high. The ICC indicated strong agreement (ICC[A,10]=0.903, 95% CI 0.869‐0.930; P<.001), indicating strong consistency in the application of the 10 evaluation criteria across evaluators.

GPT-4 evaluations indicated that the text generated by ChatGPT-4o using a step-by-step approach combined with the uploaded context (prompt 4) had comparatively higher scores for readability and scientific soundness among the evaluated outputs, which was broadly consistent with patterns observed in the human evaluations. By contrast, prompt 1 (zero-shot without the uploaded context) tended to receive lower ratings (Multimedia Appendix 6). The personalized GPT evaluations simulating a medical doctor and a family member showed substantial agreement (weighted κ coefficient: +0.818).

For the LLM models, outputs generated using LLaMA 3 70B and Mistral tended to receive comparatively higher scores in domains such as scientific or clinical validity, accuracy, and readability. Among the prompt categories, prompt 2 (zero-shot with the uploaded context) appeared to show comparatively higher scores across multiple dimensions, whereas prompt 1 (zero-shot without the uploaded context) showed lower scores. Outputs generated using the uploaded context tended to receive higher scores for scientific validity and lower scores for omissions. Similarly, the use of prompt engineering strategies tended to be associated with higher scores for accuracy and readability and lower difficulty-in-reading ratings (Figure 3).

Although some variability was observed across age groups, female evaluators tended to assign slightly higher scores than male evaluators. Physicians tended to provide marginally lower overall scores than nonmedical evaluators. No clear differences in evaluation scores were observed according to direct involvement in the development of the PICS guidelines (Multimedia Appendix 7).

Variability between human and AI evaluations was observed, particularly for the highest- and lowest-ranked texts. Weighted κ coefficients suggested low-to-negative agreement (human vs GPT simulating a medical doctor −0.423; human vs GPT simulating a family member −0.276; Figure 4). Nevertheless, texts that were ranked highly by both human and LLM evaluators tended to have been generated using a step-by-step approach combined with text-augmented prompting, although the specific LLM models differed.

The 9 texts selected through the screening process were among the higher-scoring outputs based on the LLM-based evaluation of all 72 generated texts (Multimedia Appendices 8-10). In mixed-effects models including all 72 texts (Multimedia Appendices 11 and 12), Gemma, MedLLaMA, and Meditron:7b showed significantly lower mean scores compared with LLaMA3 70B in both patient- and physician-simulated evaluations. ChatGPT-4o and Mistral did not show statistically significant differences. Prompt engineering strategies were not significantly associated with scores in these models. Sensitivity analyses using mean scores across the three repeated outputs demonstrated similar directional trends (Multimedia Appendices 13 and 14).

In this study, information leaflets on PICS generated by commercially available LLMs for patients and their families were acceptable in terms of feasibility and validity. Among the 9 selected texts, those created using prompt engineering and text-augmented prompting approach tended to receive comparatively higher scores for readability, scientific accuracy, and overall quality, as assessed by human and AI evaluators. Subgroup analyses suggested that nonmedical evaluators tended to assign higher scores. Notably, the evaluation results from GPT-4 did not align with the human ratings. These findings should be interpreted as descriptive, given the limited number of texts in each subgroup.

A key strength of this study is that assessments were performed by nonmedical individuals and multidisciplinary health care professionals, including physicians with expertise in relevant fields. Although the information leaflets were intended for patients and their families, validation by medical professionals is essential to ensure content accuracy and appropriateness. Another notable strength of this study was the use of commercially available LLMs with text-augmented prompting approach to generate these materials. Although these models typically lack domain-specific knowledge, constructing a specialized LLM requires substantial resources. Our findings suggest that commercially available LLMs can produce content of satisfactory quality, thereby potentially facilitating their integration into clinical settings.

This study highlights the critical influence of LLM selection on the quality of information leaflets related to PICS. Given the diversity of their training datasets, LLMs inherently produce variable output quality [19,20]. A previous study evaluated the performance of 4 LLMs in generating therapeutic recommendations across 3 medical specialties and revealed significant differences in quality, accuracy, and potential harm. Notably, GPT-4 demonstrated a high level of alignment with physician assessments and proved to be effective in automating these evaluations [21]. Another study reported that GPT-4 outperformed other LLMs and human responses in generating accurate, relevant, helpful, and safe answers to patient questions regarding laboratory test results [22]. In our study, the LLaMA 3 70B and Mistral models tended to produce comparatively higher-quality outputs, which may be an important factor influencing performance.

In addition, we explored the impact of prompt engineering using strategies, including zero-shot, few-shot, and step-by-step approaches. Prompt engineering strategies can significantly affect the accuracy, integrity, and overall quality of the generated content [23]. Our results demonstrated that texts generated using simplistic zero-shot prompts without the uploaded context tended to receive lower scores across all evaluation criteria. In contrast, the step-by-step approach, which was previously reported to improve the output quality [24], tended to show comparatively higher scores. Moreover, the integration of retrieval-augmented generation has been reported to enhance factual accuracy and reduce content omissions [10]. Integrating structured guideline reformatting and advanced prompt engineering strategies using the GPT-4 Turbo significantly improved the accuracy of clinical decision support in chronic hepatitis C management, with the guideline context and formatting demonstrating more effectiveness than few-shot learning [25]. Taken together, these findings suggest the potential importance of using structured prompts and the integration of external knowledge retrieval in optimizing LLM performance in health care communications.

Subgroup analyses revealed significant differences in the evaluation scores based on evaluator characteristics. Female evaluators typically assigned higher scores than their male counterparts, which may reflect potential gender-related differences in the perception of readability or clarity [26]. In addition, physicians consistently rated the texts lower than nonmedical evaluators, which may reflect stricter professional standards or a more critical appraisal of clinical accuracy and completeness [27]. No clear differences in evaluation scores were observed according to direct involvement in the development of the PICS guidelines. This may support the validity and objectivity of the proposed evaluation methodology. The documents generated using the LLM were more acceptable to nonmedical individuals, suggesting that these texts may be relatively easier to use for communication with patients and their families. However, owing to the variation in ratings among evaluators, it is important to consider that perceptions may differ depending on the medical experience of the patients or their families.

Interevaluator variability analyses highlighted key differences between human- and AI-based assessments. Although both human and AI evaluators tended to rank outputs generated using step-by-step prompting combined with text-augmented prompting approach more favorably, variability was observed in the assessment of texts at extremes of performance, particularly in terms of scientific validity, readability, and overall appropriateness. Notably, the use of prompts simulating a family member improved the agreement between the LLMs and human experts, although the degree of alignment varied across the subscales. This observation suggests that the complexity of the evaluation content and the quality of the generated outputs can influence agreement levels, emphasizing the importance of careful prompt engineering and caution when relying solely on AI-based assessments [28]. Similar trends have been observed in clinical evaluations. For instance, when comparing ChatGPT’s assessment of clinical cases with evaluations by 2 board-certified physicians, only moderate interevaluator reliability was achieved, with notable discrepancies in domains such as diagnosis and treatment decisions, highlighting the risk of overreliance on AI in sensitive medical contexts [29]. Future studies should further investigate the factors contributing to human-AI discrepancies and optimize methodologies to enhance the alignment between AI-generated evaluations and expert human judgments.

Effective communication is a key determinant of patient satisfaction. Patient satisfaction is strongly associated with the quality of physician-patient communication and that of discharge planning, thus emphasizing the importance of interpersonal engagement in clinical care [30]. In recent years, patients have increasingly turned to the internet and digital media to obtain health-related information independently [31]. Although such information-seeking behavior empowers patients, it introduces new communication challenges for clinicians, who must address misinformation and convey accurate explanations effectively under limited timeframes. Integrating commercially available LLMs into health care communication workflows may help bridge this gap by enabling patients to interact with reliable, understandable, and personalized explanations, potentially enhancing their engagement and satisfaction.

This study has several limitations.

First, all the generated texts focused exclusively on PICS, limiting the generalizability of the findings to other clinical subjects.

Second, a deliberate design choice in this study was to prioritize real-world feasibility over technical optimization. Accordingly, we used default generation parameters and a simple document-upload approach rather than implementing a fully indexed retrieval-augmented generation pipeline, reflecting how health care professionals are most likely to use commercially available LLMs in routine clinical practice. Furthermore, the use of a default 2048-token context window represents a technical limitation. We did not verify whether the full text of the uploaded guidelines was retained in the context window during generation. Consequently, partial truncation may have occurred, potentially attenuating the benefits of context-augmented prompting. While this limits the internal validity of the models’ theoretical maximum performance, it accurately reflects real-world deployment conditions, where clinicians typically rely on default system settings without control over context management. In addition, the instruction to “search” in prompt 4 was semantically ambiguous given the offline environment. While our functional intent was to enforce reliance on the uploaded context rather than pretrained knowledge, the explicit use of the term “search”—instead of “extract” or “summarize”—may have inadvertently triggered retrieval-simulation behaviors or role-playing. Since the generated texts were evaluated based on this prompt, we cannot rule out the possibility that this phrasing influenced the stylistic presentation or narrative structure of the outputs. This discrepancy represents a methodological limitation regarding prompt design.

Third, no formal sample size calculation was performed because this was a pilot study intended to explore preliminary trends. The human evaluation represents a comparative assessment among outputs, meeting a minimal threshold of completeness rather than the full distribution of all generated texts. The selection process was not statistically driven but was designed to ensure the feasibility of human evaluation. In addition, the selection procedures, including the use of the longest output as a proxy for informational coverage and the expert-based filtering prior to evaluation, may have preferentially retained more complete or higher-quality texts while excluding less suitable outputs. As a result, the evaluated sample may not fully represent the distribution of all generated texts and may have led to an overestimation of overall quality. Therefore, the findings should not be interpreted as estimates of overall model performance, but rather as exploratory comparisons across model-prompt configurations under feasibility-constrained conditions. Moreover, the final human evaluation was based on 9 selected texts. Certain prompts (prompts 2 and 3) were represented by only a single text (n=1). Such small and uneven group sizes substantially limit statistical power and the reliability of detecting meaningful differences across prompt strategies. Therefore, these comparisons are exploratory and descriptive and do not allow definitive conclusions regarding the superiority of any specific prompt configuration.

Fourth, the number of human evaluators was relatively small, which may have affected the representativeness of the findings. Although interevaluator reliability was formally assessed using an ICC, residual variability across evaluators and evaluation items may still have remained. Furthermore, discrepancies were observed between human and LLM-based evaluations; however, the limited sample size made it difficult to systematically examine the structural factors underlying these differences. Future studies using larger datasets are needed to investigate the mechanisms driving divergence between human and LLM assessments and to develop AI-based evaluation methods that more accurately reflect human judgment patterns.

Fifth, LLM outputs were generated in English and subsequently translated into Japanese using a GPT-4–based system prior to evaluation. Accordingly, human evaluators assessed the translated Japanese texts rather than the original English outputs. Although translations were performed without formal human postediting, prior studies suggest that GPT-4 achieves translation quality comparable to that of junior professional translators [17], supporting the general adequacy of this approach. However, subtle linguistic or cultural nuances may not have been fully preserved. In addition, the GPT-4–based translation process may have systematically normalized sentence structure and wording, potentially increasing readability- and clarity-related scores, independent of the intrinsic quality of the source LLM outputs. Therefore, evaluation scores—particularly for domains such as readability, clarity, and difficulty of reading—may reflect not only the quality of the LLM-generated content but also the characteristics of the translation process, which may complicate the attribution of differences to individual models or prompt strategies. In this context, at the time of the study, several commercially available LLMs exhibited more stable performance in English than in Japanese, which motivated the decision to generate outputs in English prior to translation. While this approach reflects a pragmatic real-world workflow, it introduces an additional processing layer that may affect the interpretability. With recent advances in multilingual capabilities and agent-based control of LLMs, direct generation in Japanese has become increasingly feasible. Future studies should therefore consider evaluating outputs generated natively in the target language to better isolate the effects of model architecture and prompt design. In addition, because safety-related domains consistently received high scores, their inclusion in the composite score may have obscured differences in quality-related scores. To enhance transparency, safety and quality domains were additionally presented separately in the figure.

Finally, patients and their family members were not included in the evaluation process, which may limit the real-world applicability of our findings. For clinical translation, incorporating direct feedback from a target audience would provide stronger evidence of clinical effectiveness.

Among the selected texts included in the final human evaluation, informational materials generated using commercially available LLMs were generally rated as acceptable by human evaluators, and none contained harmful content. These findings suggest that LLMs may support the development of patient-facing informational materials under feasibility-constrained conditions, although further validation with larger and more diverse samples is warranted.