This paper is in the following e-collection/theme issue:

Early Results in Infodemiology and Infoveillance (72) Medicine 2.0: Social Media, Open, Participatory, Collaborative Medicine (1641) Infodemiology and Infoveillance (1349) Quality/Credibility of eHealth Information and Trust Issues (531) Information Quality in Digital Media (340) Diabetes (187)

Published on in Vol 8 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/57720, first published .
Oral Diabetes Medication Videos on Douyin: Analysis of Information Quality and User Comment Attitudes

Oral Diabetes Medication Videos on Douyin: Analysis of Information Quality and User Comment Attitudes

Oral Diabetes Medication Videos on Douyin: Analysis of Information Quality and User Comment Attitudes

Authors of this article:

Baolu Zhang1, 2, 3 Author Orcid Image ;   Surintorn Kalampakorn3 Author Orcid Image ;   Arpaporn Powwattana3 Author Orcid Image ;   Jutatip Sillabutra4 Author Orcid Image ;   Gang Liu5 Author Orcid Image
Article Authors Cited by Tweetations Metrics

    Original Paper

    1Department of Nursing, The Affiliated Hospital, Southwest Medical University, Luzhou, China

    2School of Nursing, Southwest Medical University, Luzhou, China

    3Department of Public Health Nursing, Faculty of Public Health, Mahidol University, Bangkok, Thailand

    4Department of Biostatistics, Faculty of Public Health, Mahidol University, Bangkok, Thailand

    5Department of Orthopedics and Center for Orthopedic Diseases Research, The Traditional Chinese Medicine Affiliated Hospital, Southwest Medical University, Luzhou, China

    *these authors contributed equally

    Corresponding Author:

    Surintorn Kalampakorn, PhD

    Department of Public Health Nursing

    Faculty of Public Health

    Mahidol University

    420/1 Rajavithi Rd

    Bangkok, 10400

    Thailand

    Phone: 66 96 646 2696

    Email: surintorn.kal@mahidol.ac.th


    Background: Oral diabetes medications are important for glucose management in people with diabetes. Although there are many health-related videos on Douyin (the Chinese version of TikTok), the quality of information and the effects on user comment attitudes are unclear.

    Objective: The purpose of this study was to analyze the quality of information and user comment attitudes related to oral diabetes medication videos on Douyin.

    Methods: The key phrase “oral diabetes medications” was used to search Douyin on July 24, 2023, and the final samples included 138 videos. The basic information in the videos and the content of user comments were captured using Python. Each video was assigned a sentiment category based on the predominant positive, neutral, or negative attitude, as analyzed using the Weiciyun website. Two independent raters assessed the video content and information quality using the DISCERN (a tool for assessing health information quality) and PEMAT-A/V (Patient Education Materials Assessment Tool for Audiovisual Materials) instruments.

    Results: Doctors were the main source of the videos (136/138, 98.6%). The overall information quality of the videos was acceptable (median 3, IQR 1). Videos on Douyin showed relatively high understandability (median 75%, IQR 16.6%) but poor actionability (median 66.7%, IQR 48%). Most content on oral diabetes medications on Douyin related to the mechanism of action (75/138, 54.3%), precautions (70/138, 50.7%), and advantages (68/138, 49.3%), with limited content on indications (19/138, 13.8%) and contraindications (14/138, 10.1%). It was found that 10.1% (14/138) of the videos contained misinformation, of which 50% (7/14) were about the method of administration. Regarding user comment attitudes, the majority of videos garnered positive comments (81/138, 58.7%), followed by neutral comments (46/138, 33.3%) and negative comments (11/138, 8%). Multinomial logistic regression revealed 2 factors influencing a positive attitude: user comment count (adjusted odds ratio [OR] 1.00, 95% CI 1.00-1.00; P=.02) and information quality of treatment choices (adjusted OR 1.49, 95% CI 1.09-2.04; P=.01).

    Conclusions: Despite most videos on Douyin being posted by doctors, with generally acceptable information quality and positive user comment attitudes, some content inaccuracies and poor actionability remain. Users show more positive attitudes toward videos with high-quality information about treatment choices. This study suggests that health care providers should ensure the accuracy and actionability of video content, enhance the information quality of treatment choices of oral diabetes medications to foster positive user attitudes, help users access accurate health information, and promote medication adherence.

    JMIR Form Res 2024;8:e57720

    doi:10.2196/57720

    Keywords

    diabetes; oral diabetes medication; information quality; user comment attitude; video analysis; Douyin 


    Diabetes has emerged as a major public health challenge worldwide. As of 2021, around 537 million adults worldwide were diagnosed with diabetes, with projections indicating an increase to 783 million by 2045 [1]. China has the largest number of people with diabetes, with 140 million people constituting 26.2% of the worldwide diabetes population [1]. Notably, 96% of these cases are of type 2 diabetes mellitus (T2DM) [2]. A study conducted between 2007 and 2017 in China revealed an increase in diabetes awareness from 39.4% to 53.6%. However, the rates of treatment and glycemic control were suboptimal, standing at 32.5% and 32.8%, respectively [3]. Effective glycemic control is crucial in preventing diabetes-related microvascular and macrovascular complications [4]. Therefore, it is imperative to take action to enhance glycemic control in patients with diabetes.

    Pharmacotherapy, including insulin therapy and oral medications, remains the predominant method of glycemic control, in addition to lifestyle modifications [5]. Research indicates that oral medications are more effective in helping patients, particularly those with diabetes who have a disease duration shorter than 5 years, achieve and maintain stable blood glucose levels compared to insulin therapy [6]. Although oral medications are crucial, nonadherence to medications remains a significant challenge. Medication nonadherence rates among patients with T2DM range from 21.2% to 59.8% [7-9]. This issue may be linked to limited health literacy among patients [10]. Improving health literacy through health education is essential for addressing this problem [11]. The provision of medication information is recognized as a fundamental aspect of diabetes health education [12]. Research indicates that 56.7% of patients with diabetes express information needs regarding oral medications, ranking second after glycemic control–related information [13]. Nevertheless, patients usually encounter difficulty in finding medical information related to their condition that is easy to understand [12]. This challenge highlights the need for more accessible and engaging forms of health education.

    In recent years, social media has emerged as a promising platform for delivering health information to the public [14]. Among various social media platforms in China, Douyin (the Chinese version of TikTok), which is dedicated to the creation and sharing of short videos, is increasingly being used by doctors to publish public health education clips and is attracting many users searching for health information. Despite some differences in visual effects, privacy settings, the market, and government regulation, TikTok and Douyin are, at face value, the same [15]. The latest data report on Douyin's scientific outreach [16] reveals that content related to health care is among the most viewed by its users. By 2022, the platform hosted 35,000 certified doctors, producing over 4.43 million scientific educational content videos. Notably, diabetes ranks as 1 of the most frequently searched diseases.

    Nevertheless, the informational quality of Douyin's short videos varies significantly, leading to the proliferation of misinformation. Evidence suggests that although qualified organizations and other individual users, such as health professionals and science communicators, provide information based on evidence, the material provided by for-profit firms is of poorer quality [17-19]. The presence of misinformation in health-related content on social media platforms is a growing concern, as it could potentially lead to inaccurate guidance or even health risks for users [20]. Although a study indicated that the quality of diabetes-related videos on Douyin is generally acceptable, with a primary focus on disease management, some aspects, such as the quality and comprehensiveness of information regarding oral diabetes medications, are not adequately covered, and the existing health information still does not fully meet the needs of patients [17]. Given that oral medication is a crucial component of diabetes self-management [21], assessing the quality of information and identifying potential misinformation in videos related to oral diabetes medications become imperative to meet the needs and preferences of patients and users and to ensure the dissemination of accurate health information on these platforms.

    Although, like other video-sharing social platforms, the number of likes on TikTok (English version of Douyin) is an important indicator of user preferences [22], user comments, as a vital channel for interaction and communication with creators, more effectively express user views and emotional attitudes, as they encompass text, emojis, and other forms of expression [23]. Current research focusing on user preferences in video content predominantly uses likes as the key indicator for evaluation. For instance, Zhang et al [24] determined users' positive attitudes based on the number of likes a video received and found that content related to treatment, signs and symptoms, and social and cultural aspects in YouTube diabetes videos received the highest popularity. However, this approach often neglects the sentiment and emotions expressed in user comments, which offer valuable insights into viewers' genuine opinions. Analyzing user comments can provide a more comprehensive understanding of user preferences and help identify areas for improvement in video content creation and recommendation. To summarize, the informational quality of short videos on Douyin regarding oral diabetes medications is not well established, and research on user comment attitudes, as reflected in comments, is limited. Therefore, to bridge this gap, this study aimed to assess the information quality, content, and attitudes expressed in user comments on Douyin videos related to oral diabetes medications and explore the factors that influence these user comment attitudes.


    Search Strategy and Data Extraction

    This study conducted a basic search (simply entering the search term without additional filters) on the Douyin platform using the Chinese term “糖尿病口服药” (which means “oral diabetes medications”) on July 24, 2023, and selected the first 200 video web links delivered by Douyin's comprehensive ranking as our sample. The sample size was determined using the interval or ratio calculation formula [25], while considering the consistency levels of the DISCERN instrument (a tool for assessing health information quality) [26]. The calculated minimum sample size was 120, but considering the potential for invalid samples and referring to similar previous studies [17,18], 200 samples were ultimately determined. We chose to conduct the basic search on a single day to ensure consistency in the available video pool, given the platform's daily content updates. By using comprehensive ranking rather than focusing solely on the most popular or most recent videos, this method encompassed a broader range of content, potentially mitigating biases related to popularity or recency. Although search results may be influenced by Douyin's personalized recommendations for different users, this approach reflects typical user behavior when searching for health information. Notably, this sampling approach has been widely used in previous Douyin video analysis studies [17,19].

    All web links containing information from July 25-27, 2023, were extracted. Videos directly related to oral medicines for T2DM were included. Exclusion criteria were repeated videos; videos irrelevant to T2DM and oral medication; no narration, as narration is a key feature of educational videos that effectively delivers information through the combined use of audio and visual elements and videos lacking narration may not serve as comprehensive educational materials despite containing relevant visual information; and videos no longer available on the platform (taken off shelves), as their content and user comments could not be accessed for analysis. After the screening, we collected 138 (69%) of 200 videos for additional data extraction and analysis (Figure 1).

    ‎
    Figure 1. Flowchart of the screening process for T2DM-related videos on the Douyin platform from July 24 to 27, 2023. T2DM: type 2 diabetes mellitus.

    Evaluation Methodology

    Three aspects of oral diabetes medication-related videos on Douyin were evaluated: information quality, video content, and user comment attitudes. First, we used Python to extract video characteristics (days posted, length, number of likes, number of shares, user comments, and publisher replies to comments). Next, the main researcher manually reviewed the videos and associated publisher profiles on Douyin to collect publisher characteristics (source [doctor vs news agency], gender, department, and professional title). These characteristics were obtained from the publishers' self-reported information on their official account homepages. All this information was then organized in Microsoft Excel.

    Second, we assessed the information quality using 2 instruments, the DISCERN and the PEMAT-A/V (Patient Education Materials Assessment Tool for Audiovisual Materials) [27]. DISCERN has been widely used to evaluate the quality of health information in Douyin short videos [17-19]. The instrument consists of 16 questions in 3 sections: reliability of the publication (questions 1-8), treatment choices (questions 9-15), and overall quality of the publication (question 16). Responses are rated using a 5-point Likert scale, ranging from 1 (poor) to 5 (good) [26]. PEMAT-A/V has been used as an instrument to evaluate the comprehensibility and actionability of patient education A/V materials [28]. The instrument consists of 17 questions in 2 sections: understandability (questions 1-13) and actionability (questions 14-17). The scores for the responses are 0 or 1, reflecting disagreement or agreement, respectively. The final score (%) equals (total points/total possible points) ×100. In PEMAT-A/V, a score of ≥70% in either understandability or actionability indicates that a particular aspect is considered good, while a score of <70% in any area means it is considered poor [28].

    Third, we conducted content analysis and identified potential misinformation in the videos. The oral diabetes medication video content was classified into 9 aspects: mechanisms of action, indications, methods of administration, adverse effects, contraindications, efficacy, precautions, advantages, and importance based on the recent guidelines for oral hydrological medications [29] and the common themes identified by researchers during the review of video content. All 9 aspects were set as dichotomous variables (ie, mentioned vs not mentioned). The specific codebook is shown in Multimedia Appendix 1, which provides definitions and examples of each of the 9 aspects. During this content analysis process, the researchers also identified potential misinformation, which was defined as any content that contradicted or was inconsistent with the guidelines used for content classification. A third assessor (a Chinese endocrinology specialist) was consulted to verify the identified misinformation and proposed clarifications.

    Furthermore, we implemented a rigorous rating process to ensure the reliability of our analysis. Two researchers independently worked on information quality assessment, content analysis, and identification of misinformation. Interrater reliability was evaluated using IBM SPSS Statistics 18.0 software. The interrater reliability ranges from 0.813 to 0.861 for 16 items in the DISCERN instrument, from 0.867 to 0.901 for 17 items in the PEMAT-A/V instrument, from 0.801 to 0.834 for each of the 9 aspects of medication video content, and from 0.811 to 0.821 for misinformation that pertains to video material. All reliability coefficients are statistically significant at the 0.1% error margin level. These results indicate satisfactory interrater reliability.

    Finally, sentiment analysis of user comments was conducted using the Weiciyun website [30] to determine the proportion of attitudes (positive, neutral, and negative) for each video. The Weiciyun website, an online tool specialized in Chinese text analysis, leverages rule-based and machine learning algorithms to accurately identify sentiment tendencies in Chinese texts, while effectively handling context ambiguities. Compared to general sentiment attitude analysis tools, Weiciyun offers advantages in processing Chinese texts, incorporating advanced techniques for improved analysis quality [30]. Moreover, considering the large volume of comments to be analyzed, manual analysis would have been impractical and prone to human error. In contrast, the Weiciyun website enables efficient and code-free data analysis, often used in conjunction with Python, making it more suitable for this study.

    The predominant attitude of video comments, determined by the highest percentage response (>50%), was used to categorize each video. The specific steps involved (1) importing user comments from each Python-crawled video; (2) incorporating custom sentiment words and modifying emoticon settings, such as adding words like “like,” “heart,” “applauding,” and “thanks” for positive sentiments and words like “sobbing,” “cracking up,” and “awkwardly laughing” for negative attitudes; (3) automatically analyzing the proportion of attitude in each video comment (positive, neutral, negative); and (4) assigning an attitude category based on the highest percentage. If in a video, no single category had a percentage >50% or the percentages were equally distributed among the 3 categories, the video was assigned a neutral attitude.

    Statistical Analysis

    Data were analyzed using IBM SPSS Statistics 18.0 software. Categorical data were described as frequencies (%), while continuous variables were presented as the median (IQR) due to their nonnormal distribution. Chi-square tests and Kruskal-Wallis tests were performed for categorical and continuous variable comparisons, respectively, with Bonferroni adjustment for all pairwise comparisons. Variables with significant associations (P<.05) in these analyses were included in a multinomial logistic regression analysis to investigate the association between each video's basic characteristics, the quality of information (evaluated using DISCERN and PEMAT-A/V scores), and 9 content-related aspects with user comment attitudes.

    Ethical Considerations

    This study analyzed publicly available videos on oral diabetes medications and their associated user comments on Douyin. All data collected were anonymized and deidentified. As the research did not involve direct human participation and only used public information, it was exempt from ethical review by the Mahidol University Institutional Review Board.


    Basic Characteristics and User Comment Attitudes in Videos

    Table 1 shows the characteristics of oral diabetes medication videos on Douyin categorized by user comment attitude. In this study, 138 Douyin videos related to oral diabetes medications were analyzed. Regarding publisher characteristics, it was noted that most of these videos were uploaded by doctors on individual accounts (136/138, 98.6%). Conversely, the news agencies of organizational accounts contributed the least number of videos (2/138, 1.4%). The gender distribution of the publishers was relatively balanced, with males representing 54.3% (75/138) and females 45.7% (63/138). The majority of publishers were affiliated with the internal medicine department (115/138, 83.3%), while a smaller proportion were pharmacists (11/138, 8%). Most of the publishers held a senior professional title, serving as either the chief or deputy chief of their departments (113/138, 81.9%). In terms of video characteristics, the collected videos were posted on Douyin for a median duration of 217.5 (IQR 376) days. These videos were 70 (IQR 55) seconds long and received a median of 1897 (IQR 7957) likes, 385.5 (IQR 2400) shares, and 76.5 (IQR 242) comments. Notably, the median number of publisher replies was 0 (IQR 4). Regarding user comment attitudes, the majority of videos garnered positive comments (81/138, 58.7%), followed by neutral comments (46/138, 33.3%) and negative comments (11/138, 8%). The Kruskal-Wallis test results indicated significant differences in the number of user comments across videos with various comment attitudes (P=.002). Videos that received predominantly negative and neutral comments had significantly higher numbers of user comments compared to those that received predominantly positive comments (both P<.05).

    Table 1. Characteristics of oral diabetes medication videos on Douyin categorized by user comment attitude.
    VariablesOverall (N=138)Positive comments (n=81)Neutral comments (n=46)Negative comments (n=11)Statistical testP value
    Publisher source, n (%)a

    		Doctors136 (98.6)80 (98.8)45 (97.8)11 (100.0)0.357.840

    		News agencies2 (1.4)1 (1.2)1 (2.2)0—b—
    Publisher gender, n (%)a

    		Male75 (54.3)45 (55.6)25 (54.3)5 (45.5)0.398.820

    		Female63 (45.7)36 (44.4)21 (45.6)6 (54.5)——
    Publisher department, n (%)a

    		Internal medicine115 (83.3)65 (80.2)40 (87.0)10 (90.9)5.086.530

    		Surgery6 (4.3)4 (4.9)2 (4.3)0——

    		Pharmacy11 (8.0)6 (7.4)4 (8.7)1 (9.1)——

    		Others6 (4.3)6 (7.4)00——
    Publisher professional title, n (%)a

    		Chief physician80 (58.0)43 (53.1)30 (65.2)7 (63.6)2.757.840

    		Deputy chief physician33 (23.9)22 (27.2)9 (19.6)2 (18.2)——

    		Attending physician14 (10.1)10 (12.3)3 (6.5)1 (9.1)——

    		Pharmacist11 (8.0)6 (7.4)4 (8.7)1 (9.1)——
    Video characteristics,median (IQR)c

    		Days posted217.5 (376)250 (429)196.5 (293)173 (414)0.868.650

    		Length (seconds)70 (55)71 (58)70 (54)59 (50)0.101.950

    		Likes1897 (7957)1152(5511)3724.5 (11461)5623 (17578)4.652.100

    		Shares385.5 (2400)216 (1491)839 (6333)2231 (7165)5.594.060

    		User comments76.5 (242)38 (161)150.5 (472)d564 (698)d12.707.002

    		Publisher replies with comments0 (4)0 (4)0 (7)0 (1)1.468 .480

    aChi-square test.

    bNot applicable.

    cKruskal-Wallis test.

    dSignificant differences from the positive-attitude group: neutral (P=.01) and negative (P=.03).

    Information Quality

    Table 2 shows the information quality of oral diabetes medication videos on Douyin assessed by DISCERN and PEMAT-A/V and categorized by user comment attitude. Our findings indicated that the median overall information quality was 3 (IQR 1) out of a maximum score of 5 on the DISCERN instrument, and the median understandability was 75% (IQR 16.6%) for oral diabetes medication videos on Douyin. Both scores were considered acceptable. However, the median actionability was rated as poor, at 66.7% (IQR 48.0%). Kruskal-Wallis test results showed that the videos' reliability, information quality of treatment choices, total DISCERN scores, and PEMAT understandability scores exhibited significant differences across different user comment attitude types (P=.002, P=.001, P<.001, and P=.02, respectively). Furthermore, multiple comparison analysis showed that videos receiving predominantly positive comments had a significantly higher information quality of treatment choices and total DISCERN scores compared to those receiving neutral and negative comments and a significantly higher reliability and understandability compared to those receiving neutral comments (all P<.05).

    Table 2. Information quality of oral diabetes medication videos on Douyin assessed by DISCERN and PEMAT-A/Va and categorized by user comment attitude.
    VariablesOverall (N=138), median (IQR)Positive comments (n=81), median (IQR)Neutral comments (n=46), median (IQR)Negative comments (n=11), median (IQR)Statistical testP value
    DISCERNb

    		Reliability of videos (questions 1-8)25 (4)26 (4)24 (6)c23 (4)12.47.002

    		Information quality of treatment choices (questions 9-15)15 (4)16 (4)14.50 (4)c13 (3)c13.42.001

    		Overall information quality (question 16)3 (1)3 (1)3 (1)3 (0)4.19.120

    		Total DISCERN score44 (6)44 (5)41 (8)c40 (9)c19.04<.001
    PEMAT-A/Vb

    		Understandability75% (16.6%)77.8% (19.9%)70% (18.6%)c70% (16.6%)7.59.020

    		Actionability66.7% (48.0%)66.7% (66.7%)50% (33.4%)66.7% (16.7%)3.36.190

    aPEMAT-A/V: Patient Education Materials Assessment Tool for Audiovisual Materials.

    bKruskal-Wallis test.

    cSignificant differences from the positive-attitude group: reliability of videos (neutral P=.003); quality of treatment choices (neutral P=.006, negative P=.03); total DISCERN scores (neutral P<.001, negative P=.04); PEMAT-A/V understandability (neutral P=.03).

    Video Content

    In terms of video content, Table 3 shows that among 138 videos, the mechanism of action was the most frequently mentioned theme with 75 (54.3%) videos, followed by precautions with 70 (50.7%) videos and advantages with 68 (49.3%) videos. In contrast, indications and contraindications were rarely mentioned, accounting for 19 (13.8%) and 14 (10.1%) videos, respectively. Moreover, chi-square test results demonstrated that there were statistically significant differences in the proportions of videos mentioning 2 content themes (mechanisms of action and advantages) across 3 types of user comment attitudes (P<.001 and P=.001, respectively). Additionally, multiple comparison analysis revealed that videos receiving predominantly positive comments mentioned these 2 themes at a significantly higher rate than those receiving neutral comments, and the difference was significant (P<.05). Regarding misinformation in video content, our results indicated that 10.1% (14/138) of the videos contained misinformation (Table 3). Furthermore, these 14 (10.1%) videos spanned 5 content themes (mechanisms of action, precautions, methods of administration, efficiency, and importance). It should be noted that the majority of the videos with misinformation (7/14, 50%) contained misinformation regarding methods of administration (Table 4).

    Table 3. Content of oral diabetes medication videos on Douyin categorized by user comment attitude.
    VariablesOverall (N=138), n (%)Positive comments (n=81), n (%)Neutral comments (n=46), n (%)Negative comments (n=11), n (%)Statistical testP value
    Video contenta

    		Mechanism of action75 (54.3)55 (67.9)14 (30.4)b6 (54.5)16.60<.001

    		Precautions70 (50.7)48 (59.3)18 (39.1)4 (36.4)5.74.060

    		Advantages68 (49.3)30 (37.0)33 (71.7)b5 (45.5)14.21.001

    		Efficacy64 (46.4)37 (45.7)22 (47.8)5 (45.5)0.06.970

    		Methods of administration54 (39.1)34 (42.0)17 (37.0)3 (27.3)1.02.600

    		Adverse effects48 (34.8)32 (39.5)12 (26.1)4 (36.4)2.34.310

    		Importance27 (19.6)14 (17.3)12 (26.1)1 (9.1)2.28.320

    		Indications19 (13.8)15 (18.5)3 (6.5)1 (9.1)3.78.150

    		Contraindications14 (10.1)9 (11.1)4 (8.7)1 (9.1)0.20.900
    Misinformationa14 (10.1)8 (9.9)4 (8.7)2 (18.2)0.89.640

    aChi-square test.

    bSignificant differences from the positive-attitude group: mechanisms of action (neutral P<.001); advantages (neutral P<.001).

    Table 4. Misinformation and clarifications identified in oral diabetes medication videos on Douyin (N=14).
    Content theme and misinformationVideos, n (%)Clarifications
    Mechanisms of action

    		Glinides are insulin sensitizers.1 (7.1)Erroneous. Glinides are insulinotropic agents.
    Precautions

    		Do not make up for a missed dose of glinide medicines after a meal.2 (14.3)Too absolute. Adjust the dose of glinide medicines between meals based on blood glucose levels; there is no need to make up the dosage if it is close to the next meal.
    Methods ofadministration

    		Metformin is taken with or after a meal.6 (42.9)Too general. Regular and extended-release metformin tablets are taken with and after meals, while enteric-coated tablets are taken before meals.

    		Take sulfonylurea medicines once or twice a day.1 (7.1)Too general. Take short-acting sulfonylurea medicines 3 times a day and medium- to long-acting ones once or twice daily.
    Efficacy

    		Glinides only control blood glucose levels after meals.1 (7.1)Too absolute. Glinides also have some effect on fasting blood glucose levels.

    		SGLT2a inhibitors reduce postprandial blood glucose levels.1 (7.1)Too limited. SGLT2 inhibitors can reduce fasting and postprandial blood glucose levels.
    Importance

    		SGLT2 inhibitors and GLP-1b receptor agonists have replaced insulin, sulfonylureas, and glargine as first-line hypoglycemic agents and should be chosen first.1 (7.1)Controversial.

    		Simeglutide can completely replace insulin.1 (7.1)Controversial.

    aSGLT2: sodium-glucose transporter 2.

    bGLP-1: glucagon-like peptide-1.

    Factors Affecting User Comment Attitudes

    Although the dependent variable (user comment attitudes: positive, neutral, negative) was ordinal, the independent variables did not pass the parallel line test (P<.001), prompting the use of multinomial logistic regression analysis in this study. Additionally, 10 independent variables from Tables 1, 2, and 3 with P<.05 were subjected to collinearity testing. It was found that except for “total DISCERN scores,” there was no collinearity among the remaining variables (tolerance>0.1 or variance inflation factor [VIF]<10). Consequently, 6 independent variables were ultimately included in the multinomial logistic regression analysis, and Table 5 presents the standardized results (P<.05).

    Table 5. Multinomial logistic regression analysis results of factors associated with user comment attitudes toward oral diabetes medication videos on Douyin.
    Dependent and independent variablesb (SE)WaldAdjusted ORa (95% CI)P value
    Positive commentsb

    		Comment count–0.00 (0.00)5.211.00 (1.00-1.00).020

    		Information quality of treatment choices0.40 (0.16)6.381.49 (1.09-2.04).010
    Neutral commentsc

    		Information quality of treatment choices–0.26 (0.10)7.110.77 (0.64-0.93).010

    		Advantages (not mentioned)–1.41 (0.48)8.730.24 (0.10-0.62).003

    		Mechanism of action (not mentioned)1.14 (0.46)6.153.12 (1.27-7.66).010
    Negative commentsb

    		Comment count0.00 (0.00)5.211.00 (1.00-1.00).020

    		Information quality of treatment choices–0.40 (0.16)6.380.67(0.49-0.91).010

    aOR: odds ratio.

    bUsing the negative category as a reference.

    cUsing the positive category as a reference.

    In multinomial logistic regression analysis, influence factors, such as the number of user comments (P=.02) and the quality of information regarding treatment choices (P=.01), were found to be associated with videos receiving predominantly positive comments. However, videos receiving positive comments had an odds ratio (OR) of 1.00 (95% CI 1.00-1.00) for the number of user comments, indicating lower odds compared to videos receiving negative comments, which had an OR of 1.00 (95% CI 1.00-1.00). Furthermore, the OR for having high-quality information regarding treatment choices was 1.49 (95% CI 1.09-2.04) for videos receiving positive comments, while videos receiving negative comments had an OR of 0.67 (95% CI 0.49-0.91). However, videos receiving neutral comments had ORs of 0.77 (95% CI 0.64-0.93) for the information quality of treatment choices, 0.24 (95% CI 0.10-0.62) for the proportion of unmentioned medication advantages, and 3.12 (95% CI 1.27-7.66) for the proportion of unmentioned mechanisms of action (Table 5).


    Principal Findings

    Our study explored the quality of information and user comment attitudes toward oral diabetes medication videos on Douyin. We found that most of the videos were uploaded by doctors, with generally acceptable information quality and received predominantly positive user comments. Notably, videos with higher information quality tended to receive more positive user comments. However, a minority of these videos contained misinformation, and the actionability of the video needs to be improved. Additionally, there was limited focus on the indications and contraindications of medications.

    Our study determined that the quality of information in Douyin videos concerning oral medications for diabetes is generally acceptable [17]. This finding is similar to those for most other diseases, such as myopia [18] and chronic obstructive pulmonary disease [19], which were posted by health professionals. This may be ascribed to our findings that 98.6% of the videos on oral diabetes medications in our study originated from senior doctors' personal accounts. This was paired with a relatively low rate of misleading information, at only 10.9%, a notable contrast to other topics. For instance, YouTube videos on T2DM have a misleading information rate of 32.4% [31], while those on prostate cancer have a misleading information rate of 41.2% [32]; attention-deficit/hyperactivity disorder, 52% [33]; and pediatric urological diseases, as high as 77.8% [34]. Nevertheless, the occurrence of misinformation in videos disseminated by senior doctors underscores the necessity for continual improvement. Prior studies have confirmed that misleading online information can have harmful effects [35]. For instance, Weitzman et al [36] suggested that misleading information regarding diabetes cures prompts skepticism about their effectiveness. Therefore, our study suggests that health care professionals are crucial in maintaining the quality of health information, while also emphasizing the need to further ensure the accuracy of the health information they disseminate.

    Unfortunately, the actionability of oral diabetes medication–related videos was found to be unsatisfactory. This lack of practical applicability is a concern, as patients often seek such videos for guidance on managing their health conditions. Our findings are not isolated; similar issues with actionability have been documented in online health information across various topics [32,33,37,38]. For instance, Xu et al [32] indicated that health videos on prostate cancer also frequently fall short in providing actionable steps. Zhang et al [39] further supported this, highlighting a general trend in online health content where the focus on providing clear, actionable advice is often lacking. These observations underline a critical gap in digital health communication, emphasizing the need for content creators, particularly health professionals, to prioritize actionability in their materials. As highlighted by Hilfiker et al [40], when using digital health information for decision-making, it is critical to focus on specific instructions and action steps. Therefore, we suggest that professor-level physicians contribute more patient-centered, high-quality videos on oral diabetes medication, emphasizing actionable content to provide patients with brief and clear instructions on necessary steps.

    Regarding video content, our study noted that the majority of videos were about mechanisms of action, precautions, advancements, efficiency, and methods of administration, but only a few videos fully covered other aspects, such as indications and contraindications. Previous research has demonstrated the significant influence of digital health information on patients' decision-making [41]. When individuals search for information about diabetes medications, they often turn to online sources to determine the suitability of those medications for their condition and any potential contraindications. Consequently, a deficiency in clear and easily accessible information concerning indications and contraindications can result in misunderstandings and potentially harmful decisions. Furthermore, Douyin's official platform has recognized this issue and has required creators to specify indications and contraindications details when discussing treatments or medication [42]. Therefore, we suggest that health care professionals improve the provision of information concerning the indications and contraindications of oral diabetes medications to support patients in making well-informed health decisions.

    This study suggested that the majority of user comments on oral diabetes medication–related videos on Douyin exhibit a positive comment attitude. In comparison to negative attitudes, the information quality of treatment choices in videos positively influenced users' attitudes. Previous studies have found that accurate, high-quality, well-structured, and informative content is essential for encouraging positive user engagement [43,44]. This study further revealed the importance of the information quality of treatment choices, including medication therapy principles, benefits and risks, consequences of nonuse, impact on individuals and families, and support for shared decision-making, positively influencing users' attitudes and reactions.

    Moreover, an intriguing aspect of our findings is that the videos receiving predominantly positive comments were associated with significantly fewer user comments. The reason might be attributed to typical user behaviors and interaction patterns on social media, as well as the influence of individual personality differences. Prior studies have confirmed that users often prefer to observe and consume content silently on social media [45], and posts presented in video format or posts about rational content (informational, functional, and educational) do not significantly encourage active user comments [46,47]. Additionally, personality traits also affect online commenting behavior. Disagreeable individuals are more likely to comment to voice dissent, while neurotic users tend to comment more on emotionally impactful content [48]. Therefore, videos eliciting positive sentiment may suppress commenting driven by negative motivations (eg, disagreement or emotional distress) among these groups. Echoing these findings, our study revealed that users, upon viewing satisfying content regarding oral diabetes medication–related videos, consider it unnecessary to further comment.

    It is worth mentioning that our study suggested there is no significant association between the length of the videos, the video duration, the publisher's gender, their professional title, and user comment attitudes toward Douyin videos related to oral diabetes medications. Previous studies on YouTube videos [24,49] have also confirmed that the video duration does not impact user comment attitudes. However, Zhang et al [24] found that the gender of publishers and the duration of posts significantly impact user comment attitudes toward diabetes-related video content on YouTube. This inconsistency may be attributed to the different video formats across platforms. Douyin videos are generally shorter than YouTube videos, requiring a shorter attention span, which might lead to different user engagement patterns. Our findings suggest focusing on the content and quality of oral diabetes medication–related videos on Douyin to help patients obtain accurate health information and promote medication adherence.

    Limitations and Future Directions

    The findings of this study should be considered in the light of a few limitations. The videos analyzed in this study were in Chinese, which may limit the generalizability of the results to TikTok videos on oral diabetes medications in other languages. Future studies should assess the content quality of these videos in various languages for a more comprehensive understanding. Additionally, the selection of the top 200 videos recommended by Douyin's algorithm through a single-account search may introduce limitations, as search results can vary across user accounts due to personalized recommendations. Future research could explore multiaccount searches or longitudinal sampling to capture a broader content range and gain insights into the algorithm's functioning. Furthermore, the sentiment analysis conducted by the Weiciyun website may not fully capture the nuances and context of user comments due to language variability and algorithmic constraints, potentially leading to inaccuracies. It is recommended for future research to use more advanced sentiment analysis tools and algorithms to improve accuracy. Moreover, categorizing sentiment based on the highest percentage may oversimplify the distribution, particularly when sentiment is evenly spread across categories, leading to an incomplete representation of user comment attitudes. Future research could explore alternative methods for sentiment classification. Nonetheless, this study represents 1 of the first steps in advancing research involving the relationship between the information quality of Douyin oral diabetes medication videos and user comment attitudes, which goes beyond the previous studies limited to the broader spectrum of diabetes.

    Conclusion

    Our study assessed the quality of information in 138 Douyin videos on oral diabetes medications. Results indicated that most videos, predominantly posted by doctors, generally offer acceptable information quality, focusing on mechanisms of action, precautions, and advantages. However, they lack comprehensive details on medication indications and contraindications, with some misinformation and dissatisfied actionability. User comment attitudes toward these videos are predominantly positive, influenced by the information quality of treatment choices. We conclude that patients should be cautious when seeking diabetes medication information on Douyin. We recommend that health care providers improve the accuracy and usefulness of their content to better inform treatment choices and enhance user confidence. Furthermore, we advise Douyin to refine its medical content review process.

    Acknowledgments

    The study was supported by the Sichuan Modern Design and Cultural Research Center in China (MD22E006) and the Southwest Medical University Social Science Association Research Project (SMUSS202227).

    Data Availability

    The data sets generated and analyzed during the study are not publicly available due to privacy concerns and content sensitivity. User comments and creator information pose identification risks, while health-related content about diabetes medications requires a proper context to avoid misinterpretation. The data sets are available from the corresponding author upon reasonable request.

    Authors' Contributions

    The authors declare that no artificial intelligence tools, including large language models, such as ChatGPT, were used in any part of the manuscript writing process, including ideation, content generation, or editing.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Codebook for hand-coding oral diabetes medication-related Douyin videos.

    DOCX File , 23 KB
    1. IDF diabetes atlas 2021. International Diabetes Federation. URL: https://diabetesatlas.org/atlas/tenth-edition/ [accessed 2024-02-16]
    2. Ong KL, Stafford LK, McLaughlin SA, Boyko EJ, Vollset SE, Smith AE, et al. [CrossRef] [Medline]
    3. Jin C, Lai Y, Li Y, Teng D, Yang W, Teng W, et al. China National Diabetes and Metabolic Disorders Study Group and the Thyroid Disorders, Iodine Status and Diabetes Epidemiological Survey Group. Changes in the prevalence of diabetes and control of risk factors for diabetes among Chinese adults from 2007 to 2017: an analysis of repeated national cross-sectional surveys. J Diabetes. Mar 05, 2024;16(2):e13492. [FREE Full text] [CrossRef] [Medline]
    4. Ishiguro H, Kodama S, Horikawa C, Fujihara K, Hirose AS, Hirasawa R, et al. In search of the ideal resistance training program to improve glycemic control and its indication for patients with type 2 diabetes mellitus: a systematic review and meta-analysis. Sports Med. Jan 24, 2016;46(1):67-77. [CrossRef] [Medline]
    5. Zhao R, Lu Z, Yang J, Zhang L, Li Y, Zhang X. Drug delivery system in the treatment of diabetes mellitus. Front Bioeng Biotechnol. Dec 2020;8(1):880. [FREE Full text] [CrossRef] [Medline]
    6. Zuo P, Shi J, Yan J, Yang L, Liu C, Yuan L, et al. Effects of insulin therapy and oral hypoglycemic agents on glycemic control for type 2 diabetes mellitus patients in China-a case control study. Exp Clin Endocrinol Diabetes. May 08, 2021;129(5):374-378. [CrossRef] [Medline]
    7. Wu X, Yang H, Yuan R, Long E, Tong R. Predictive models of medication non-adherence risks of patients with T2D based on multiple machine learning algorithms. BMJ Open Diabetes Res Care. Mar 09, 2020;8(1):e001055. [FREE Full text] [CrossRef] [Medline]
    8. Raum E, Krämer HU, Rüter G, Rothenbacher D, Rosemann T, Szecsenyi J, et al. Medication non-adherence and poor glycaemic control in patients with type 2 diabetes mellitus. Diabetes Res Clin Pract. Sep 2012;97(3):377-384. [CrossRef] [Medline]
    9. Zhang ZP, Premikha M, Luo M, Venkataraman K. Diabetes distress and peripheral neuropathy are associated with medication non-adherence in individuals with type 2 diabetes in primary care. Acta Diabetol. Mar 19, 2021;58(3):309-317. [CrossRef] [Medline]
    10. Capoccia K, Odegard PS, Letassy N. Medication adherence with diabetes medication: a systematic review of the literature. Diabetes Educ. Feb 2016;42(1):34-71. [CrossRef] [Medline]
    11. Heine M, Lategan F, Erasmus M, Lombaard C, Mc Carthy N, Olivier J, et al. Health education interventions to promote health literacy in adults with selected non-communicable diseases living in low-to-middle income countries: a systematic review and meta-analysis. J Eval Clin Pract. Dec 22, 2021;27(6):1417-1428. [CrossRef] [Medline]
    12. Lee DYL, Armour C, Krass I. The development and evaluation of written medicines information for type 2 diabetes. Health Educ Res. Dec 08, 2007;22(6):918-930. [CrossRef] [Medline]
    13. Weymann N, Härter M, Dirmaier J. Information and decision support needs in patients with type 2 diabetes. Health Inform J. Mar 10, 2016;22(1):46-59. [FREE Full text] [CrossRef] [Medline]
    14. Kanchan S, Gaidhane A. Social media role and its impact on public health: a narrative review. Cureus. Jan 2023;15(1):e33737. [FREE Full text] [CrossRef] [Medline]
    15. Kaye DBV, Chen X, Zeng J. The co-evolution of two Chinese mobile short video apps: parallel platformization of Douyin and TikTok. Mob Media Commun. Aug 30, 2020;9(2):229-253. [CrossRef]
    16. Douyin Healthcare Assistant. Douyin health science popularization data report. Douyin. 2023. URL: https://www.douyin.com/note/7219193337868340535 [accessed 2024-02-16]
    17. Kong W, Song S, Zhao YC, Zhu Q, Sha L. TikTok as a health information source: assessment of the quality of information in diabetes-related videos. J Med Internet Res. Sep 01, 2021;23(9):e30409. [FREE Full text] [CrossRef] [Medline]
    18. Ming S, Han J, Li M, Liu Y, Xie K, Lei B. TikTok and adolescent vision health: content and information quality assessment of the top short videos related to myopia. Front Public Health. Jan 4, 2022;10:1068582. [FREE Full text] [CrossRef] [Medline]
    19. Song S, Xue X, Zhao YC, Li J, Zhu Q, Zhao M. Short-video apps as a health information source for chronic obstructive pulmonary disease: information quality assessment of TikTok videos. J Med Internet Res. Dec 20, 2021;23(12):e28318. [FREE Full text] [CrossRef] [Medline]
    20. Song S, Zhang Y, Yu B. Interventions to support consumer evaluation of online health information credibility: a scoping review. Int J Med Inform. Jan 2021;145:104321. [CrossRef] [Medline]
    21. Zhang Y, Fu Y, Mu YM, Huang Y, Xuan J. Network meta-analysis of the therapeutic effects of hypoglycemic drugs and intensive lifestyle modification on impaired glucose tolerance. Clin Ther. Sep 2021;43(9):1524-1556. [FREE Full text] [CrossRef] [Medline]
    22. How TikTok recommends videos #ForYou. TikTok. 2020. URL: https://newsroom.tiktok.com/en-us/how-tiktok-recommends-videos-for-you [accessed 2024-02-16]
    23. Cheng Z, Li Y. Like, comment, and share on TikTok: exploring the effect of sentiment and second-person view on the user engagement with TikTok news videos. Soc Sci Comput Rev. May 25, 2023;42(1):201-223. [CrossRef]
    24. Zhang J, Zheng Z, Wang Y, Zhu Y. A study on users' preference towards diabetes-related video clips on YouTube. BMC Med Inform Decis Mak. Feb 28, 2020;20(1):43. [FREE Full text] [CrossRef] [Medline]
    25. Kachigan S. Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods. New York. Radius Press; 1986.
    26. Rees CE, Ford JE, Sheard CE. Evaluating the reliability of DISCERN: a tool for assessing the quality of written patient information on treatment choices. Patient Educ Couns. Jul 2002;47(3):273-275. [CrossRef] [Medline]
    27. Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: a systematic review of indicators, criteria, tools, and evaluation results. J Assoc Inf Sci Technol. Apr 29, 2015;66(10):2071-2084. [CrossRef]
    28. Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. Sep 2014;96(3):395-403. [FREE Full text] [CrossRef] [Medline]
    29. Ganesan K, Rana M, Sultan S. Oral hypoglycemic medications. In: StatPearls. Treasure Island, FL. StatPearls Publishing; 2023.
    30. Weiciyun: simple and powerful word cloud text cloud word cloud art generator. Weiciyun. URL: https://www.weiciyun.com/ [accessed 2024-02-16]
    31. Leong AY, Sanghera R, Jhajj J, Desai N, Jammu BS, Makowsky MJ. Is YouTube useful as a source of health information for adults with type 2 diabetes? A South Asian perspective. Can J Diabetes. Aug 2018;42(4):395-403.e4. [CrossRef] [Medline]
    32. Xu AJ, Taylor J, Gao T, Mihalcea R, Perez-Rosas V, Loeb S. TikTok and prostate cancer: misinformation and quality of information using validated questionnaires. BJU Int. Oct 21, 2021;128(4):435-437. [CrossRef] [Medline]
    33. Yeung A, Ng E, Abi-Jaoude E. TikTok and attention-deficit/hyperactivity disorder: a cross-sectional study of social media content quality. Can J Psychiatry. Dec 2022;67(12):899-906. [FREE Full text] [CrossRef] [Medline]
    34. O'Sullivan NJ, Nason G, Manecksha RP, O'Kelly F. The unintentional spread of misinformation on 'TikTok'; a paediatric urological perspective. J Pediatr Urol. Jun 2022;18(3):371-375. [CrossRef] [Medline]
    35. Swire-Thompson B, Lazer D. Public health and online misinformation: challenges and recommendations. Annu Rev Public Health. Apr 02, 2020;41(1):433-451. [FREE Full text] [CrossRef] [Medline]
    36. Weitzman ER, Cole E, Kaci L, Mandl KD. Social but safe? Quality and safety of diabetes-related online social networks. J Am Med Inform Assoc. May 01, 2011;18(3):292-297. [FREE Full text] [CrossRef] [Medline]
    37. Lopez Ramos C, Williams JE, Bababekov YJ, Chang DC, Carter BS, Jones PS. Assessing the understandability and actionability of online neurosurgical patient education materials. World Neurosurg. Oct 2019;130:e588-e597. [CrossRef] [Medline]
    38. Gulbrandsen MT, O'Reilly OC, Gao B, Cannon D, Jesurajan J, Gulbrandsen TR, et al. Health literacy in rotator cuff repair: a quantitative assessment of the understandability of online patient education material. JSES Int. Nov 2023;7(6):2344-2348. [FREE Full text] [CrossRef] [Medline]
    39. Zhang Z, Lu Y, Kou Y, Wu D, Huh-Yoo J, He Z. Stud Health Technol Inform. Aug 21, 2019;264:1403-1407. [FREE Full text] [CrossRef] [Medline]
    40. Hilfiker S, Santana S, Freedman M, Harris L. There's a gap between digital health information and users - let's close it. Stud Health Technol Inform. Jun 25, 2020;269:324-331. [CrossRef] [Medline]
    41. Thapa DK, Visentin DC, Kornhaber R, West S, Cleary M. The influence of online health information on health decisions: a systematic review. Patient Educ Couns. Apr 2021;104(4):770-784. [CrossRef] [Medline]
    42. TikTok Healthcare Assistant. Guidelines for medical content review on TikTok. TikTok. Apr 2, 2023. URL: https://tinyurl.com/bdd7nuhx [accessed 2024-02-16]
    43. Cinelli M, Peruzzi A, Schmidt AL, Villa R, Costa E, Quattrociocchi W, et al. Promoting engagement with quality communication in social media. PLoS One. Oct 13, 2022;17(10):e0275534. [FREE Full text] [CrossRef] [Medline]
    44. Lee J, Hong IB. Predicting positive user responses to social media advertising: the roles of emotional appeal, informativeness, and creativity. Int J Inf Manag. Jun 2016;36(3):360-373. [CrossRef]
    45. Gong W, Lim E, Zhu F. Characterizing silent users in social media communities. 2021. Presented at: ICWSM-15: Ninth International AAAI Conference on Web and Social Media; May 26-29, 2015:140-149; Oxford, UK. [CrossRef]
    46. Sabate F, Berbegal-Mirabent J, Cañabate A, Lebherz PR. Factors influencing popularity of branded content in Facebook fan pages. Eur Manag J. Dec 2014;32(6):1001-1011. [CrossRef]
    47. Dolan R, Conduit J, Frethey-Bentham C, Fahy J, Goodman S. Social media engagement behavior: A framework for engaging customers through social media content. EJM. Oct 07, 2019;53(10):2213-2243. [CrossRef]
    48. Barnes R, Mahar D, Cockshaw W, Wong I. Personality and online news commenting behaviours: uncovering the characteristics of those below the line. Media Int Aust. Sep 20, 2018;169(1):117-130. [CrossRef]
    49. Gabarron E, Fernandez-Luque L, Armayones M, Lau AY. Identifying measures used for assessing quality of YouTube videos with patient health information: a review of current literature. Interact J Med Res. Feb 28, 2013;2(1):e6. [FREE Full text] [CrossRef] [Medline]

    ‎
    A/V: audiovisual
    GLP-1: glucagon-like peptide-1
    OR: odds ratio
    PEMAT-A/V: Patient Education Materials Assessment Tool for Audiovisual Materials
    SGLT2: sodium-glucose transporter 2
    T2DM: type 2 diabetes mellitus

    Edited by A Mavragani; submitted 25.02.24; peer-reviewed by Y Yao, H Crocket; comments to author 01.05.24; revised version received 17.08.24; accepted 10.09.24; published 18.10.24.

    Copyright

    ©Baolu Zhang, Surintorn Kalampakorn, Arpaporn Powwattana, Jutatip Sillabutra, Gang Liu. Originally published in JMIR Formative Research (https://formative.jmir.org), 18.10.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.