This study aimed to compare the accuracy of rumor identification and the effectiveness of health science popularization between GPT-4 and ERNIE Bot 4.0, which did not involve the recruitment of human participants. We used health-related rumors, misconceptions, and truths sourced from publicly accessible web-based platforms to test GPT-4 and ERNIE Bot 4.0. The requirement for a formal ethical review was waived. In this study, all researchers, data collectors, and evaluators were aware of the research objectives and conducted the research in accordance with the study design. This study posed no harm to them.

The health-related rumors, misconceptions, and truths were selected from the China Internet Disinformation Platform (CIDP). The inclusive materials were issued from January 1 to December 10, 2023. Based on previous proof-of-concept studies, a sample size of 30 is required for feasibility testing [15,16]. Considering the primary objective was to investigate the identification capabilities, the ratio of rumors and misconceptions to truths was 2:1. By using a 2:1 ratio, we aimed to ensure that rumors, which are often more prevalent and varied in web-based environments, were adequately represented in our sample. Considering practical constraints such as time and resource limitations, the 2:1 ratio allowed us to manage these constraints while still achieving a representative and meaningful dataset for our research objectives. The rumors and misconceptions were from the “Internet Rumor Exposure Station” of the CIDP. In total, 20 health-related rumors and misconceptions were randomly selected using the random number table method. The truths were from “True Knowledge” of the CIDP, an expert-led video platform for debunking rumors. We randomly chose 10 health-related videos. To ensure accuracy, information extraction was completed through discussion by 3 PhD students (YL, YM, and YZ). All samples were randomly inputted into the 2 models in Chinese. Before inputting, the questioning was standardized through group discussion within the research team (YL, YM, and YZ). The questions were conducted in that order. The study process is presented in Figure 1. All the above work has been completed on December 25‐26, 2023. All materials and questions are shown in detail in Multimedia Appendix 1.

SPSS (version 26.0; IBM Corp) was adopted to establish the cross-checking database by 2 reviewers (YL and YM) independently. Continuous data were expressed as mean and SD, and categorical data were expressed as frequency or percentage. According to the P-P plot test, the readability and applicability scores showed an approximately normal distribution. A 2-tailed t test was adopted to compare the differences in readability scores and applicability scores between the 2 models, and a chi-square test was used to compare the applicability grade. Further, a rank-sum test was selected to compare grade distribution differences among all items of the CSAM between the 2 models. Meanwhile, textual data (health essays) were qualitatively analyzed to further illustrate the differences between items. In addition, we selected the inductive content analysis [23] to summarize the characteristics of the health essays.

According to related studies of the 2 GLLMs, asking questions in Chinese and English or repeating the same question twice may yield different answers [24,25]. We further made postanalysis validation to analyze the stability of the 2 GLLMs. If the first response of GPT-4 is inaccurate, we will rephrase the question in English and ask again. Meanwhile, we repeated the same questions for both models to evaluate reproducibility. According to the related studies [26,27], depending on the size of the dataset, 10%‐25% of data units would be typical. Although there is little consensus regarding the proportion of the data, we selected 25% of cases (5 rumors, n=20 and 3 truths, n=10).

For GPT-4, the accuracy rates were 100% (20/20) for identifying rumors or misconceptions and 70% (7/10) for truths. Meanwhile, ERNIE Bot 4.0 achieved 100% accuracy rates for both rumors or misconceptions (20/20) and truths (10/10).

Among the cases not correctly identified, GPT-4 demonstrates hallucinations, revealed as inconsistencies between explanations and judgments [28,29]. For another example, “Drinking milk regularly can still lead to Vitamin D deficiency” is a truth. The outputs of GPT-4 demonstrated that drinking milk alone might not be enough to meet the body’s needs, but it thought this information was a rumor. Similarly, inaccuracies occurred in the identification of other truths.

Among the explanations, GPT-4 and ERNIE Bot 4.0 mostly provided widely recognized viewpoints. It is worth noting that the 2 models may provide criteria or indicators of health. For example, GPT-4 recommends an average daily water intake of about 2‐3 L for adults without any additional restrictions. ERNIE Bot 4.0 suggests that in mild climates, persons with light levels of physical activity need to drink 1500-1700 mL of plain water per day. In high temperatures or high levels of physical activity, a moderate increase in water intake is required. In addition, drinking water should be evenly distributed throughout the day to avoid excessive drinking at one time. Furthermore, it also stated that these recommendations come from the Chinese Dietary Guidelines, while the GPT-4 did not provide reference information. We further verified that the citation by ERNIE Bot 4.0 followed the recommendations of the guideline [30]. In response to more specific questions, some suggestions were vague and unclear. For the explanation of “Diabetic patients can eat fruit,” GPT-4 only provided a general principle—controlling blood sugar levels—while ERNIE Bot 4.0 provided more specific requirements—smooth control of postprandial blood glucose within 10 mmol/L—which is in compliance with guidelines [31,32].

The readability scores of 30 health essays are presented in Table 1. The average R value for the health essays was 2.92 (SD 0.85) in GPT-4 and 3.02 (SD 0.84) in ERNIE Bot 4.0. There was no statistically significant difference between the 2 models (P=.65). For the total number of words, ERNIE Bot 4.0 produced longer essays than GPT-4 (P=.03). For the average sentence length, ERNIE Bot 4.0’s sentences were shorter than GPT-4’s (P<.01). For comparisons of rumors or misconceptions and truths, respectively, please see Tables S1 and S2 in Multimedia Appendix 2.

The CSAM scores of 30 health essays are presented in Table 2, and the grade is presented in Table 3. Except for the content and cultural appropriateness, significant differences were observed in the total score and scores in other dimensions between the 2 GLLMs (P<.05). Generally, the scores of ERNIE Bot 4.0 were higher than those of GPT-4 except for layout or typography. Among the CSAM grades, significant differences were only found in the learning stimulation dimension (P=.003). The item score grade distribution is presented in Table 4 and Multimedia Appendix 3. Significant grade distribution differences were observed in items b3, b5, d3, and e2 (P<.05).

Considering the potential impact of language [13,24], 2 weeks later after the first questioning, we asked GPT-4 in English and received its English response for the truths not correctly identified. However, GPT-4 still responded that “Drinking milk regularly can still lead to Vitamin D deficiency” was a rumor, despite the English explanations were accurate. For other truths, GPT-4 still replied “it is a topic of debate.”

When the same question is submitted twice, it is a possibility that the 2 responses generated may not be similar before and after the second submission [25]. Further, we asked the same questions to the 2 models again from January 23 to 24, 2024. Notably, GPT-4 expressed differential opinions in its 2 responses for the same case (“Eating more spinach for iron supplements”). In the first time, GPT-4 argued that the viewpoint was inaccurate, whereas, in the second time, it considered the view correct. The 2 explanations were consistent, both agreeing that the viewpoint was theoretically feasible but had little effect in practice. Overall, among other cases, there was consistency in the 2 time-point responses with some differences in detail. For instance, about a truth—“You can drink milk on an empty stomach”—in comparison with the first response, GPT-4 added an explanation about food culture.

In the second response, ERNIE Bot 4.0 accurately identified the rumor or misconception, and while the 2 explanations were similar overall, the second one was longer and included more details. For the truth that “You can drink milk on an empty stomach,” the second response provided additional information, covering special situations like lactose intolerance. When it came to the rumor that “Sunlight through glass windows can replenish calcium,” the second response not only explained the cause but also offered dietary advice. Interestingly, in debunking the rumor that “Drinking more water is healthier,” the first response mentioned the recommendation of Chinese Dietary Guidelines, but it disappeared in the second response. This inconsistency might suggest randomness in their responses.

When comparing GPT-4 and ERNIE Bot 4.0, both performed well in identifying rumors or misconceptions, but GPT-4 demonstrated hallucinations. Notably, both models provided comprehensive explanations. However, they rarely provided supporting citations for their viewpoints. On the CIDP, without proven scientific evidence, it has been recognized as a rumor and could mislead the public. The unconfirmed information is shared on the web as if it were accurate information, sometimes going viral and causing widespread anxiety or leading to misinformation-based beliefs [34].

For recommendations and viewpoints provided by the 2 GLLMs, we further asked them to offer references or citations. Regrettably, the 2 models indicated that they were unable to provide accurate source material, but they sometimes could provide vague references. Upon further verification, we found that the references cited as evidence were either full of mistakes or completely fabricated. This is consistent with previous studies conducted on earlier versions of ChatGPT [35,36]. Our findings suggest that the same problem still persists on GPT-4 and ERNIE Bot 4.0. Especially, the provided numerical values should be carefully examined. The value may be from nonauthoritative reports [25]. Nevertheless, the 2 models usually provide widely recognized viewpoints, and the recommended healthy lifestyles are also aligned with the guidelines. Overall, despite limitations, they can achieve the effectiveness of science popularization for the general public in the situation that there is no health care professional to judge information.

Readability is a critical factor in comprehension, which refers to how well a passage of text can be read and understood by an individual [37]. For low literacy skills, a fifth-grade level or lower is more suitable [38]. Obviously, the essays generated by the 2 GLLMs met the requirements in reading level (Table 1). In our study, although the readability scores of the 2 GLLMs were not statistically different, ERNIE Bot 4.0 generated longer text than GPT-4 (Table 1). For an individual with lower literacy skills, longer text may lead to loss of interest and even cause reading difficulties. The related studies have proved that GLLMs can provide simplified text, but understanding this text still requires a certain level of education [39-41]. Additionally, reading fluency is an essential aspect of the overall reading experience. The grammatical issues in GPT-4 cannot be overlooked, and GPT-4 tends to generate longer sentences than ERNIE Bot 4.0 (P<.001; Table 1). This suggests that texts by ERNIE Bot 4.0 may be easier to read. Nevertheless, text easiness showed no influence on trustworthiness, and the public may still blindly believe this content [42]. Considering the randomness of the responses provided by the 2 models, caution should be made when taking their advice.

ERNIE Bot 4.0 was better than GPT-4 in item b3 (context is given first; P=.02) and b5 (road signs; P=.02; Table 4 and Multimedia Appendix 3). The context is given before new information. We learn new facts or behaviors more quickly when told the context first. Meanwhile, headers or topic captions tell very briefly what is coming next. These “road signs” make the text look less intimidating and prepare the reader’s thought process to expect the announced topic [38]. GPT-4 performed better than ERNIE Bot 4.0 in item d3 (subheadings used; P=.01; Table 4 and Multimedia Appendix 3). Longer texts need to be partitioned into smaller chunks. Subheadings can help readers quickly understand the core content of each chunk [38]. In item e2 (behaviors are modeled and specific), ERNIE Bot 4.0 performed better than GPT-4 (P=.004; Table 4 and Multimedia Appendix 3). People often learn more readily when specific, familiar instances are used rather than abstract or general concepts [38]. When presented with the health essay “There is no standard for the ideal time of sleep,” GPT-4 explained more about factors affecting sleep duration. In contrast, ERNIE Bot 4.0 dedicated approximately 50% of its response to helping individuals determine the appropriate length of sleep for themselves.

One of GLLM’s strengths is its ability to sift through massive amounts of information and produce responses in a conversational and easy-to-understand manner. In contrast, search results from current search engines can often be overwhelming for the general public, frequently filled with irrelevant or misleading information [25]. However, as ChatGPT and ERNIE Bot are specifically designed for conversational dialogue, their responses can be more comprehensible than those found in professional guidelines or literature. The study showed that, after reading popularized papers addressed to a lay audience, lay readers agreed more with the knowledge claims presented in those papers and were more confident in their claim judgments than after reading papers addressed to expert readers [43]. In our study, we found that the 2 GLLMs provided easy-to-understand information. However, the information might be superficial or nonspecific with errors. Therefore, if someone agrees with the idea because of the model’s response, the misleading information spreads widely on the web possibly [44].

Another key element for health education is stimulation and motivation. Interaction, in particular, plays a crucial role in health education or popularization. We suggest that a questioning format should be used to keep the reader recalling the educational content. In the health essays provided by the 2 models, a question-and-answer format was used, which involved posing a question and then promptly providing the corresponding answer. Even though it is a passive interaction, it is still more effective in reminding readers to focus on the following answers than long explanatory or illustrative texts. Meanwhile, it is recommended that both models successfully break down complex topics. This not only makes the materials more understandable but also allows the readers to explore the logic of cause and effect. When people believe tasks and behaviors are valuable and practicable, they will be motivated to learn [38,45].

This study has some limitations. First, due to time constraints, we were unable to use all the cases in the rumor library. We chose typical recent cases to test their ability as in previous studies. Second, the readability formula is a general tool. In the future, it is worth exploring to develop the specific tool for Chinese medical materials. Third, the CSAM can help you save time and money and improve program effectiveness by selecting or producing materials that the public is likely to pick up, read, understand, and act on. The CSAM cannot substitute for formative research and testing, through which the public confirms that the information is attractive, useful, and persuasive to them. Fourth, the 2 GLLMs do not provide graphics. In many works of scientific popularization, graphic is an important tool to help readers understand. However, with the emergence of GLLMs’ capability to generate graphics, we need to explore whether this function can assist in science popularization further in depth. Fifth, this study was to compare 2 GLLMs. With more and more GLLMs being developed, future research can further explore the applicability of them in the Chinese context. Finally, this study focuses more on the Chinese context. Future exploration will be to include more linguistics experts to jointly discuss the impact of different languages on large language models.

ERNIE Bot 4.0 demonstrated similar accuracy to GPT-4 in identifying Chinese rumors. Notably, the 2 GLLMs mostly provided widely recognized viewpoints, despite potential inaccuracies and superficiality. When it comes to the health essays, nurses can learn from the linguistic strengths that facilitate readability and comprehension. However, the content needs to be checked by experts for appropriateness and accuracy. Additionally, ERNIE Bot 4.0 aligns well with Chinese expression habits, making it a suitable choice for those seeking an enhanced Chinese reading experience.