This study conducted a cross-sectional analysis of vaccine-related questions and associated information needs on Quora and Naver KiN between June 2020 and June 2021. We followed the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology; Multimedia Appendix 1) reporting guidelines. This study design is aligned with recent cross-sectional analyses of social media use for health-related issues [53].

To examine the perceptions and information needs of Q&A site users regarding the COVID-19 vaccine, we collected data from Q&A sites with representative US and Korean user bases. For this purpose, we chose Quora, a US platform hosting millions of questions and answers [54], and Naver KiN, the foremost social Q&A site in Korea [55]. Data were collected using Selenium (Software Freedom Conservancy, Inc), an open-source framework that allows automatic interaction with web browsers using Python 3.8.5 (Python Software Foundation) [56]. We set intervals during data collection to avoid overusing the service. For each question, we collected the URL, upload date, and answers. The data collection ran from June 27, 2020, to June 27, 2021, based on the upload dates for both websites, and the inclusion criteria were questions and answers pertinent to COVID-19 and any morphological form of the keyword “vaccine” (in Korean,백신), such as “vaccines” and words containing “vaccine.” By applying those criteria and excluding unanswered questions, we identified and subsequently removed questions falling outside our specified criteria.

As a result, after sampling the data, we collected 3952 sets of questions and answers from Quora and Naver KiN, as depicted in Figure 1. Rather than analyzing all the answers to the questions, we opted to examine only the most upvoted answers so that we could analyze the characteristics of the answers that users deemed most favorable [48]. Accordingly, only one answer for each question was used from an average of 27.4 and 2.7 answers detected on Quora and Naver KiN, respectively. All data and source code files can be found in the repository.

In this study, we used 2 major approaches to analyze the questions and answers (Figure 2), including topic categorization for questions based on Latent Dirichlet Allocation (LDA) [57] and Top2Vec [58], and sentiment analysis for answers using ensemble-bagging with Bidirectional Encoder Representations from Transformers (BERT) [59] and Efficiently Learning an Encoder that Classifies Token Replacement Accurately (ELECTRA) [60]. For answers, this study used the most upvoted answers to focus on the characteristics of answers that users deemed most favorable. These natural language processing (NLP) techniques have been shown to be effective for analyzing large-scale data [61].

In this study, we combined machine learning predictions with human-in-the-loop validation to categorize questions from Quora and Naver KiN. To this, we applied topic modeling to identify key topics across the documents, offering guidance for strengthening public health communication strategies [62]. Specifically, LDA was used to categorize the questions on Quora into 5 topics and those on Naver KiN into 7 topics. We initially extracted 50 topics, which is the optimal number of topics determined by the coherence score, and manually merged them into 5 topics for Quora and 7 topics for Naver KiN. Researchers (SY, SHJ, JC, and YJY) discussed and decided on each topic together. While merging topics, we discovered that Naver-KiN covered more topics than Quora.

LDA uses documents, words, and latent variables that represent the latent topics associated with word sequences. Although LDA is effective in restricting the number of topics and offering keywords associated with each topic, it does not quantify the similarity between each keyword and its topic (it mainly uses document-word frequency when categorizing topics). That makes it challenging to determine which words are most relevant to the topic when relying solely on human decisions for keyword selection. To address this limitation, we incorporate Top2Vec, which preserves word semantics by representing words and documents in a shared embedding space and derives topic representations directly from these semantic structures.

Building on this motivation, we adopt a hybrid LDA-Top2Vec pipeline to combine probabilistic topic structure with semantic embedding-based topic modeling. While Top2Vec is often applied directly to raw documents, we instead use LDA to first identify topic-relevant keywords, which serve as a coarse topical filter. This step constrains the semantic space Top2Vec explores, enabling it to focus on semantically meaningful regions associated with each topic. As a result, this hybrid approach yields more interpretable and stable topic representations than applying Top2Vec alone to the full document corpus. While formal coherence scores were not used as a selection criterion, this hybrid approach was chosen to prioritize interpretability and topic stability.

Operationally, the LDA-extracted keywords were represented as word vectors, and the questions were represented as document vectors in the vector space. Figure 2 illustrates the process of question analysis using the mixed LDA and Top2Vec method. Figure 3 further visualizes a semantic space that consists of word vectors (enclosed numbers 1, 2, 3, 4, and 5) and document vectors (labeled A, B, and C). The centroid of the document vectors is a topic vector (black central square labeled topic vector for different vaccines). The word vectors in Figure 3 are the words with the closest semantic relationship to the topic vector, whereas the LDA keywords indicate the word distribution per topic solely by frequency. In other words, LDA delineates the statistical relationship among topics, but it fails to capture semantic information about those topics [63].

Because COVID-19 greatly affected the sentiments expressed on social media [64], we used a document-level sentiment analysis to extract and classify perceptions and attitudes toward the topics discussed in the documents [32,65]. This analysis served as an indicator of public perception regarding COVID-19 concerns [66]. We used 2 transformer-based models, BERT and ELECTRA, that are capable of fine-tuning with pretrained models. The pretraining and fine-tuning mechanism enhances each model’s ability to effectively grasp the intricacies of downstream NLP tasks, such as sentiment analysis.

BERT is a widely used language model that uses pretrained data and a self-attention mechanism to generate contextualized vector outputs. It is applied to a range of downstream tasks, including sentiment analyses. BERT uses the masked language model; some tokens in the sentence are replaced with (MASK), and then the model is trained to regenerate the original tokens [59]. On the other hand, ELECTRA, which is another pretrained transformer model, uses a discriminator to predict the replaced tokens by generating token output instead of relying on a masked language model [60]. Both models have been widely used for various NLP downstream tasks [67], and we used models fine-tuned for an English sentiment analysis and a Korean sentiment analysis.

For the English language model, both BERT-based [68] and ELECTRA-based [60] were specifically fine-tuned for a sentiment analysis using Twitter datasets. While Twitter and Quora have posts of different lengths (a Twitter post consists of 8 words on average and a Quora post consists of 40 words on average) [69], both Twitter and Quora texts are social media data originating from public interactions on a social media platform. They share similar traits, and the model fine-tuned on Twitter was effectively applied to the Quora dataset [70], and we conducted a sentiment analysis of Quora data with 66% (33/50) accuracy. For the Korean language model, both BERT-based [71] and ELECTRA-based [72] were fine-tuned using Naver KiN shopping reviews, Steam game reviews, and movie reviews. We evaluated the accuracy of these models for Naver KiN data and achieved 72% (36/50) accuracy in the sentiment analysis. This choice thus mirrored the strategy we used for the English model.

One limitation of using transformer models is the constraint of the input tensor size, which is restricted to a length of 128 tokens. This limitation can sometimes introduce biases because only the first 128 tokens are considered, potentially leading to inaccurate sentiment analysis results. For instance, the beginning of a post might convey a positive sentiment by starting with a greeting, for example, that is not representative of the main content of the post. To address that issue, we divided each post into chunks of 128 tokens, measured the sentiment of each chunk, and then averaged the sentiment values of all the chunks to determine the overall sentiment result.

To further enhance the validity of the sentiment analysis, we used an ensemble method called the bagging or bootstrap aggregating technique, which is known for its effectiveness in classification tasks. This method generates an ensemble classification output that outperforms the accuracy of a single prediction value [73,74]. We applied it to predict improved sentiment values for each answer using both BERT and ELECTRA. An overview of the process is provided in Figure 4. The outcomes from both models were the probabilities of positive and negative sentiments. Subsequently, we computed the average probabilities of positive and negative sentiments. Once we obtained the averaged positive and negative probabilities, we compared them and used the higher one.

Ethical review was not required for this study because it exclusively used publicly available social media content shared in accordance with platform policies. As the data were publicly accessible, Institutional Review Board approval and a waiver of informed consent were not required [75,76]. This study did not use or infer individuals’ identities but instead examined vaccine-related opinions at an aggregate level to provide an overall picture of public perceptions. To protect user privacy, all identifiable information was removed during preprocessing, and all questions and comments presented in this manuscript were paraphrased. The final aggregated dataset contains no personally identifiable information and cannot be traced back to individual users.

Table 1 presents the topic modeling results for Quora, where 5 primary topics were identified. The most frequently discussed topic was different vaccines (978/3952, 25%), with users asking about characteristics and comparisons of various vaccine brands such as AstraZeneca (AstraZeneca plc) and Moderna (Moderna, Inc). This was followed by government policy (821/3952, 21%), where questions centered on administrative decisions and vaccine distribution, with keywords including “require,” “Trump,” and “supply.” Variants (782/3952, 20%) included discussions focused on the severity of new COVID-19 strains and vaccine efficacy. The topic of visiting overseas (743/3952, 19%) comprised questions about travel eligibility based on vaccination status. The least frequent topic, effects of vaccine (626/3952, 16%), involved concerns about immunity, side effects, and infection risks.

Table 2 shows the topic modeling outcomes for Naver KiN, which yielded 7 topics. The most common was effects of vaccine (790/3952, 20%), marked by terms like “side effects” and “uptake,” where users expressed hopes for effective vaccination and sought information on its impact. Vaccination appointments followed (711/3952, 18%), with users asking about eligibility and scheduling logistics. The variants topic (592/3952, 15%) reflected interest in vaccine efficacy against emerging strains. Government policy (553/3952, 14%) included questions about national vaccination strategies. School and education (513/3952, 13%) highlighted questions about children’s school attendance in relation to family vaccination status. Visiting overseas (474/3952, 12%) addressed international travel feasibility amid varying vaccination policies. The least common topic was different vaccines (316/3952, 8%), where users asked about vaccine suitability for individual health conditions.

A sentiment analysis was used to examine the answers given on both Quora and Naver KiN. Figure 5 illustrates the proportions of sentiments in Quora answers to questions from each topic. Across 4 of the topics—effects of vaccine, visiting overseas, variants, and different vaccines—the number of answers with negative sentiment exceeded the number of positive ones. However, within the topic of government policy, answers with positive sentiment were more prevalent than negative ones.

In Figure 5, positive versus negative sentiments were observed as follows: effects of vaccines (306/626, 48.79% vs 320/626, 51.12%), visiting overseas (335/743, 45.05% vs 408/743, 54.95%), variants (362/782, 46.29% vs 420/782, 53.71%), different vaccines (386/978, 39.47% vs 592/978, 60.53%), and government policy (437/821, 53.23% vs 384/821, 46.77%), where the first value represents positive sentiment and the second represents negative sentiment.

We analyzed sentiment outcomes on a monthly basis from June 2020 to June 2021, focusing on topic popularity over time (see Figure 6). While finer-grained temporal analyses are possible, we adopted monthly aggregation to reduce sparsity and short-term volatility and to emphasize broader discourse trends rather than reactions to specific events. Across all topics except different vaccines, positive-negative sentiment crossovers occurred at least 3 times. After May 2021, positive sentiment exceeded 50%, indicating its dominance across topics. Notably, only government policy consistently showed a surplus of positive sentiment over negative for 4 consecutive months after February 2021.

During this period, the vaccination rate increased by 15.3%, from 17.45% to 32.78% [52], coinciding with heightened public attention to vaccine rollout and access. Positive sentiment toward government policy appears to reflect evaluations of these concrete policy outcomes rather than a broader shift in institutional trust. In particular, the temporary increase in positive sentiment after February 2021 aligns with major administrative transitions and policy announcements, including changes in federal coordination, alongside continued public recognition of vaccine development efforts initiated under Trump, as reflected in the example in Table 1.

Prior research similarly suggests that during public health crises, shifts in public trust toward government are closely tied to public perception of policy responses and outcomes. Notably, policy-specific achievements (eg, visible progress in crisis management) are often associated with temporary increases in positive evaluations even amid persistent skepticism toward government institutions [77]. Thus, the observed temporal positivity in “government policy” sentiment likely captures situational approval of pandemic response measures rather than a reversal of widespread distrust in US governance.

The results of the sentiment analysis of answers to questions on Naver KiN are depicted in Figure 7. Across all 7 topics—different vaccines, government policy, schools and education, vaccine appointments, variants, effects of vaccines, and visiting overseas—positive sentiments were predominant over negative sentiments. This overarching trend is also highlighted in Figure 8, which shows that positive sentiment consistently exceeded 50% throughout the year except for the topic of different vaccines.

In Figure 7, the proportions of positive versus negative sentiment were as follows: effects of vaccines (498/790, 63.04% vs 292/790, 36.96%), visiting overseas (337/474, 71.10% vs 137/474, 28.90%), variants (367/592, 61.99% vs 225/592, 38.01%), different vaccines (218/316, 68.99% vs 98/316, 31.01%), government policy (348/553, 62.93% vs 205/553, 37.07%), vaccine appointment (562/711, 79.04% vs 149/711, 20.96%), and school and education (364/513, 70.96% vs 149/513, 29.04%).

For the topic of different vaccines, 4 positive-negative sentiment crossovers occurred between June 2020 and January 2021, often driven by concerns about side effects and individual health conditions. During this period, confirmed COVID-19 deaths per million rose sharply by 313% [52]. As shown in Table 2, a question regarding vaccine eligibility for individuals with chronic illnesses received a relevant response. After February 2021, coinciding with the launch of South Korea’s vaccination program on February 26, positive sentiment prevailed, with no further crossovers observed [52].

After analyzing the sentiments of the answers, we examined question–answer sentiment pairs within each topic to see if the question’s sentiment might affect the answer. Table 3 shows the sentiments of question-answer pairs on Quora. In 4 topics—effects of vaccine, visiting overseas, variants, and different vaccines—the predominant sentiment pairs are negative-negative, followed by negative-positive, positive-positive, and positive-negative. For the topic of government policy, negative-positive sentiment pairs are as prevalent as negative-negative, with positive-positive and positive-negative following.

Table 4 illustrates the sentiment for question-answer pairs on Naver KiN. Across all topics, positive-positive and negative-positive pairs constituted more than 63% of the sentiment. For 6 topics—effects of vaccines, visiting overseas, different vaccines, government policy, vaccine appointments, and school and education—positive-positive pairs were the most prevalent, followed by negative-positive pairs. In contrast, for the topic of variants, negative-positive pairs were the most prevalent, followed by positive-positive pairs.

Social media provides valuable insights for policymakers developing health policies [27]. Our study pioneers exploring public discourse on vaccines in 2 countries using advanced NLP techniques, surpassing traditional survey-based methods. This approach reveals hidden aspects of vaccine perceptions beyond conventional demographics. We analyzed overall perceptions (via topic analysis) and communication within each topic (via sentiment analysis). In this discussion, we consider platform characteristics and acknowledge the study’s limitations.

The United States and Korea both have concerns about 5 topics, including effects of the vaccine, visiting overseas, variants, different vaccines, and government policy; however, Korea uniquely highlights vaccine appointments and school and education. Vaccine appointments became a significant issue in Korea due to delays in vaccine distribution [49], with the government implementing an age-based distribution policy to respect the cultural emphasis on older adults and their high risk of serious disease. Although prioritizing the older adults might have effectively reduced COVID-19–related mortality and morbidity within this age group, other vulnerable populations, such as prisoners and the homeless, experienced relative disadvantages. Therefore, the equitable distribution of vaccinations is a topic requiring open discussion.

Naver KiN also revealed a unique topic about school and education that did not appear on Quora. This topic highlights how deeply education is valued in Korean culture, which made vaccinating school-aged children and their parents crucial [78]. This contrasts with the United States, where, despite the CDC’s recommendations [79], vaccine hesitancy remained a notable barrier among many parents [80]. Thus, the Naver KiN results underscore that cultural values and societal priorities, such as Korea’s strong commitment to educational success, played a significant role in shaping public attitudes toward COVID-19 vaccination.

Based on those topics, we investigated the sentiments expressed in question-answer pairs and gleaned insights into the public perceptions of COVID-19 vaccines expressed on social Q&A sites in the United States and Korea. Our findings highlight distinct sentiment patterns reflective of the cultural and informational contexts of these countries. Quora and Naver KiN both have a similar tendency of consistency between a question’s sentiment and the answers it receives (eg, positive [Q]-positive [A] or negative [Q]-negative [A]). However, users of the 2 platforms expressed different sentiments toward the COVID-19 vaccine.

The sentiment analysis of Quora data reveals that in the United States, negative sentiment was predominant in discussions about the effects of the vaccine, visiting overseas, variants, and different vaccines. In those topics, negative-negative (question-answer) pairs are the most common, indicating widespread skepticism and concern among users [15,25]. Unlike Quora, the majority of answers were positive in the Naver KiN data from Korea, with positive-positive (question-answer) and negative-positive pairs constituting more than 63% of the total. This difference may be attributed to varying levels of trust in the government between the 2 countries.

At the time we collected data from Naver and Quora, South Korea was effectively managing the pandemic, and public trust in the government was high, consistent with previous research showing high compliance with COVID-19 government policies [81]. As a result, both questions and answers were generally positive. The only topic that saw a higher occurrence of negative questions paired with answers with positive sentiment was related to variants, likely because the Korean government could not control the emergence of variants, leading to more negative sentiment. Even in discussions about variants, where there was more negative sentiment toward the question, the sentiment toward the answers remained largely positive, indicating resilience and hope [82,83]. In contrast, in the United States, the situation was markedly different. Public trust in the US government was lower, and adherence to COVID-19 policies was less consistent [81]. The skepticism and concerns expressed in the US discussions may have been driven by widespread dissatisfaction with how the pandemic was being handled, leading to a pervasive negative sentiment across various topics [81-83].

These contrasting attitudes suggest that the differences observed in the public discourse on the COVID-19 vaccine in the United States and South Korea may stem from cultural and situational factors rather than true variations in public perceptions. Given that the unique social context of each country shapes public discourse and sentiment, it can be inferred that communication strategies targeted to cultural contexts are required to address vaccine hesitancy. In the United States, public health campaigns should counter misinformation [32] and address specific safety concerns, engaging trusted community leaders and positive narratives about vaccination benefits. In Korea, on the other hand, maintaining positive public discourse through transparency and timely updates is crucial. The unique focus on vaccine appointments and education in Korean discussions highlights specific information needs that should be addressed.

Platform-specific user dynamics also play a role in shaping public discourse [84,85], affecting the sentiments of each platform’s answers. The presence of health care professionals on Naver KiN, who provide expert opinions, likely contributes to the positive sentiment and higher likelihood of their answers being perceived as trustworthy, in contrast to the diverse user base on Quora. Overall, understanding and leveraging these distinct sentiment patterns can help public health authorities craft more effective communication strategies to promote a positive and informed discourse about COVID-19 vaccination.

To select the responses for our analysis, we used a methodology intended to mirror the typical user experience on these platforms. Responses were chosen based on specific criteria designed to reflect a balance between popularity and relevance and, above all, to reflect which answers the users deemed most effective. This approach aligns with methodologies used in previous user-generated content analyses on platforms such as Quora and Naver KiN [48].

This study has several limitations. First, there was a notable variance in the average length of answers between Quora and Naver KiN. This discrepancy might have introduced potential bias when comparing the answers. Future studies could mitigate this by standardizing the length of answers or selecting questions with a similar number of words for a more meaningful comparison. In addition, platform-specific user demographics may partially confound cross-cultural comparisons. Naver KiN serves as a dominant information-seeking platform in South Korea, characterized by large-scale daily participation and broad distribution of user expertise [48,86]. In contrast, Quora users in the United States tend to be younger and more educated [87]. These structural differences may shape how questions are shaped, how answers are evaluated, and how sentiment is expressed. As a result, the observed sentiment asymmetry between the 2 platforms may reflect platform participation norms and demographic composition, as well as national-level attitudes.

Second, although we used models already trained on our target social media outlets, Quora and Naver KiN, we applied both topic modeling and sentiment analysis models without fine-tuning them with our data. This may result in less optimal and lower-performing outputs from the experiments. To mitigate this limitation, fine-tuning each model should be performed to yield more reliable results in future studies.

Third, our analysis focuses on the most favorable answers associated with each question. This design choice may limit the generalizability of findings regarding public perception. By prioritizing community-validated information, often in the form of expert-driven responses, this analysis may also introduce a positive bias toward answers. Thus, the observed association between question sentiment and answer sentiment should be interpreted as reflecting how question framing relates to the community-validated information, rather than the full range of public perception, which may include greater uncertainty, disagreement, or negative affect.

Finally, we acknowledge as a limitation that no explicit bot-detection or automated content-authenticity filtering algorithm was applied. Given the highly politicized and time-sensitive nature of vaccine-related discourse during 2020-2021, bot-generated content may have contributed to the observed sentiment patterns. Although we partially mitigated this risk by excluding unanswered questions and constraining questions that include the keyword “vaccine” (in Korean, 백신), these measures did not guarantee the removal of automated or coordinated accounts. As such, some sentiment estimates may reflect amplified or artificial signals rather than purely user opinions.

This study has provided critical insights into the dynamics of vaccine discourse, underscoring the need for context-specific strategies for promoting vaccination. By melding state-of-the-art NLP technologies with a cross-cultural analysis, we have forged a path toward more informed, efficacious, and targeted efforts in public health communication, both in the ongoing battle against the COVID-19 pandemic and beyond. Above all, this study underscores the paramount importance of harnessing social media as a prolific source of public sentiment and information needs, particularly within the sphere of vaccine hesitancy. Analyzing public discourse on platforms such as Quora and Naver KiN provides researchers with an unfiltered view of individuals’ concerns and perceptions, offering insights invaluable for crafting responsive public health strategies.

Our findings reaffirm the profound effects of cultural and societal factors on public attitudes toward vaccines. The stark contrast observed between the United States and Korea in terms of vaccine perceptions underscores the need for culturally sensitive approaches in health communication and vaccination campaigns. This study advances the academic landscape by demonstrating the efficacy of NLP technologies for comprehending public perceptions and information needs related to vaccines. NLP methods transcend traditional survey methodologies, providing a more naturalistic perspective rooted in real-world discussions on community-driven social media platforms. Our innovative amalgamation of LDA, Top2Vec, and BERT- and ELECTRA-based sentiment analyses sets a pioneering benchmark for future research in this domain. This multifaceted approach augments our comprehension of the intricacies encircling vaccine discourse and can be adapted to explore analogous public health issues across diverse countries and cultures.

As for practical insights, our findings have implications for policymakers and health communication practitioners in both countries. The discernment of disparities in vaccine-related information needs and perceptions between the United States and Korea underscores the necessity of tailoring public health campaigns to the specific nuances of each nation’s context. Effectively addressing vaccine hesitancy—a complex interplay of safety concerns, access barriers, and misinformation—demands a nuanced, multifaceted approach. Our study underscores the need for policy interventions that take these underlying factors into account, while also recognizing the profound influence of cultural and societal values on vaccination decisions.