This paper is in the following e-collection/theme issue:

Early Results from COVID-19 Studies (281) Mass Media/Social Media Communication and Campaigns (450) JMIR Theme Issue: COVID-19 Special Issue (2394) Infodemiology and Infoveillance (1278) Ethics, Privacy, and Legal Issues (458) Natural Language Processing (650) Infectious Diseases (non-STD/STI) (1490) Infoveillance, Infodemiology, Digital Disease Surveillance, Infodemic Management (1015) Vaccination and Immunization in the Digital Age (347) Discourse (38)

Published on in Vol 7 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/50367, first published .
Incivility in COVID-19 Vaccine Mandate Discourse and Moral Foundations: Natural Language Processing Approach

Incivility in COVID-19 Vaccine Mandate Discourse and Moral Foundations: Natural Language Processing Approach

Incivility in COVID-19 Vaccine Mandate Discourse and Moral Foundations: Natural Language Processing Approach

Authors of this article:

Jason Tin1 Author Orcid Image ;   Hannah Stevens2 Author Orcid Image ;   Muhammad Ehab Rasul3 Author Orcid Image ;   Laramie D Taylor3 Author Orcid Image
Article Authors Cited by Tweetations (5) Metrics

    Original Paper

    1Department of Public Health Sciences, University of California, Davis, Davis, CA, United States

    2Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Worcester, MA, United States

    3Department of Communication, University of California, Davis, Davis, CA, United States

    Corresponding Author:

    Muhammad Ehab Rasul, MA

    Department of Communication

    University of California, Davis

    1 Shields Avenue

    Davis, CA, 95616

    United States

    Phone: 1 530 752 1011

    Email: merasul@ucdavis.edu


    Background: Vaccine hesitancy poses a substantial threat to efforts to mitigate the harmful effects of the COVID-19 pandemic. To combat vaccine hesitancy, officials in the United States issued vaccine mandates, which were met with strong antivaccine discourse on social media platforms such as Reddit. The politicized and polarized nature of COVID-19 on social media has fueled uncivil discourse related to vaccine mandates, which is known to decrease confidence in COVID-19 vaccines.

    Objective: This study examines the moral foundations underlying uncivil COVID-19 vaccine discourse. Moral foundations theory poses that individuals make decisions to express approval or disapproval (ie, uncivil discourse) based on innate moral values. We examine whether moral foundations are associated with dimensions of incivility. Further, we explore whether there are any differences in the presence of incivility between the r/coronaviruscirclejerk and r/lockdownskepticism subreddits.

    Methods: Natural language processing methodologies were leveraged to analyze the moral foundations underlying uncivil discourse in 2 prominent antivaccine subreddits, r/coronaviruscirclejerk and r/lockdownskepticism. All posts and comments from both of the subreddits were collected since their inception in March 2022. This was followed by filtering the data set for key terms associated with the COVID-19 vaccine (eg, “vaccinate” and “Pfizer”) and mandates (eg, “forced” and “mandating”). These key terms were selected based on a review of existing literature and because of their salience in both of the subreddits. A 10% sample of the filtered key terms was used for the final analysis.

    Results: Findings suggested that moral foundations play a role in the psychological processes underlying uncivil vaccine mandate discourse. Specifically, we found substantial associations between all moral foundations (ie, care and harm, fairness and cheating, loyalty and betrayal, authority and subversion, and sanctity and degradation) and dimensions of incivility (ie, toxicity, insults, profanity, threat, and identity attack) except for the authority foundation. We also found statistically significant differences between r/coronaviruscirclejerk and r/lockdownskepticism for the presence of the dimensions of incivility. Specifically, the mean of identity attack, insult, toxicity, profanity, and threat in the r/lockdownskepticism subreddit was significantly lower than that in the r/coronaviruscirclejerk subreddit (P<.001).

    Conclusions: This study shows that moral foundations may play a substantial role in the presence of incivility in vaccine discourse. On the basis of the findings of the study, public health practitioners should tailor messaging by addressing the moral values underlying the concerns people may have about vaccines, which could manifest as uncivil discourse. Another way to tailor public health messaging could be to direct it to parts of social media platforms with increased uncivil discourse. By integrating moral foundations, public health messaging may increase compliance and promote civil discourse surrounding COVID-19.

    JMIR Form Res 2023;7:e50367

    doi:10.2196/50367

    Keywords

    incivility; vaccine hesitancy; moral foundations; COVID-19; vaccines; morality; social media; natural language processing; machine learning 


    Overview

    As of October 2023, more than 6,959,316 deaths and 770,875,433 cases of COVID-19 have been reported worldwide [1]. Despite the mass availability of vaccines in the United States, 32.8% of the population remains unvaccinated [2]. To increase vaccination numbers, vaccine mandates were issued across the United States. While some US adults complied with the mandates, others reacted with incivility [3]. Many public officials have caused public outrage against COVID-19 vaccines by issuing statements that minimize vaccine efficacy [4]. As such, the politicized nature of COVID-19 has increased the salience of political ideology in public health discourse [5] and sparked negative sentiment toward vaccines, which may fuel incivility toward vaccines and mandates [6,7].

    Public reliance on social media (eg, Reddit) increased heavily during the COVID-19 pandemic for various reasons, including seeking information to ease pandemic anxiety and due to social distancing—in addition to news consumption [8]. This gave rise to what scholars have coined an “infodemic” [9], where the unabated spread of COVID-19 misinformation on social media platforms undermined public trust in public health officials and their guidelines [10]. Recent work has shown that increased consumption of news related to COVID-19 leads to vaccine hesitancy and that engaging with the news on social media is linked to increased sharing and belief of COVID-19 misinformation due to various reasons, such as social media fatigue [11,12]. Uncivil vaccine discourse (eg, “Fucking Disneyland isn’t enforcing masks anymore there is no fucking reason for your college to do so especially since they are requiring the vaccine”) also decreases vaccine uptake [13,14]. Yet, the psychological mechanism underlying uncivil vaccine discourse remains unclear. Therefore, understanding the psychological processes underlying uncivil COVID-19 vaccine discourse on social media platforms is necessary to inform effective interventions.

    Incivility has been investigated across various social media platforms in political and health contexts [15-19]. Although a few studies have linked negative emotions with incivility [20], more work is needed in this area to understand the psychological processes that prompt emotion-fueled uncivil discourse. Moreover, although scholars have extensively studied COVID-19 across various contexts [3,9-11,21], very few studies have attempted to identify the theoretical underpinnings of the discourse surrounding the virus. This work fills this gap by examining the moral foundations underlying uncivil discourses on 2 prominent antivaccine subreddits by using natural language processing techniques.

    Moral foundations help individuals make decisions based on 5 innate moral values (care and harm, fairness and cheating, loyalty and betrayal, authority and subversion, and sanctity and degradation). Investigating the moral foundations of uncivil vaccine discourse can provide insight into the drivers of that incivility and offer practical implications for public health interventions against COVID-19. Thus, this work meaningfully contributes to the existing literature focused on eradicating the negative impact of COVID-19 through vaccine uptake.

    Incivility

    Definition

    Scholars across different fields have found it difficult to develop one definition of incivility. Some studies have defined incivility as impoliteness, profanity, or specific actions such as derogatory language used by political officials [16]. Coe et al [22] categorize incivility as using hateful, pejorative, or disrespectful language. Other studies have added to these definitions by including ideologically extreme arguments, exaggerated arguments, and misinformation as indicators of incivility [23-25]. Some cross-disciplinary fields conceptualize incivility as violations of norms of politeness, hostile interruptions, disrespectful behaviors, defensive reactions, and refusing to acknowledge opposing views [26-28]. We conceptualize incivility as a multidimensional construct, including toxicity, profanity, threats, insults, and discriminatory language [20].

    Moral Foundations Theory
    Overview

    The moral foundations theory (MFT) offers one explanation for vaccine discourse incivility on social media platforms. The MFT posits that individuals can adaptively make decisions and express approval or disapproval based on 5 innate moral values: care and harm, fairness and cheating, loyalty and betrayal, authority and subversion, and sanctity and degradation [29]. The care and harm dimension points out the difference between protection and the mistreatment of individuals, whereas the fairness and cheating values highlight the contrast between impartiality and dishonesty. The loyalty and betrayal values involve intergroup attachment. The authority and subversion dimension refers to the degree to which an individual follows or opposes authority. Lastly, the sanctity and degradation values focus on spirituality. Threatening individuals’ moral values can provoke uncivil behavior, such as verbally attacking vaccine proponents, to express disapproval.

    The MFT reasons that everyone shares the same core moral values. However, individuals prioritize moral values based on external factors such as cultural and environmental influences [30]. Indeed, the relative importance of individuals’ moral values is linked to compliance with COVID-19 protective measures [31-33]. An individual’s perceptions that their moral values are threatened can provoke uncivil behavior, such as verbally attacking vaccine proponents, to reassert those values; a recent study found a substantial relationship between incivility and moral foundations in social media discourse [34].

    Although researchers have used surveys to investigate moral foundations and behavior during the COVID-19 pandemic [32], such as incivility toward Asians [34], self-report data insight is limited. This work identifies salient moral foundations underlying uncivil COVID-19 vaccine discourse in an observational setting on Reddit, a social media platform that contains uncivil discourse in a naturalistic setting. Reddit provides a valuable platform to study incivility in part because it is consumed (and contributed to) by individuals worldwide (eg, in 2015, over 200 million individuals visited Reddit from 208 countries) [35]. Reddit also allows researchers to observe specific subreddits composed of individuals from certain backgrounds [36]. Therefore, we propose the following research question (RQ):

    • RQ1: What moral foundations are linked to uncivil COVID-19 discourse?
    Sanctity and Degradation

    The sanctity and degradation moral foundation refers to purity in both the spiritual and physical sense. People who value physical sanctity aim to preserve their bodily well-being. While some research has found the sanctity foundation to predict the usage of masks during the COVID-19 pandemic, other work has found that it can predict vaccine hesitancy [32,37]. One explanation is that individuals may perceive foreign substances (the vaccine, rather than a pathogen) as impure; in other words, individuals with opposing views may share a salient moral foundation [38].

    Research suggests that purity is an area of political disagreement [39]. Underpinned by cognitive dissonance theory [40,41], vaccine proponents, who may believe in vaccinations to keep their body pure from the virus, may experience cognitive discomfort in discussion with vaccine opponents, who believe vaccinations are impure, and vice versa. Researchers have argued that issues driven by values of sanctity and degradation (ie, lesbian, gay, bisexual, transgender, queer [LGBTQ] issues) result in fierce opposition because it violates a sense of purity held by anti-LGBTQ individuals [42,43]. Similarly, vaccine proponents and opponents may act uncivilly and attack each other to protect their moral perception of purity. As such, we hypothesize that:

    • H1: Purity will positively predict COVID-19 vaccine mandate incivility.
    Individualizing Foundations (Care and Fairness)

    The existing literature on MFT categorizes distinct moral foundations into two clusters: (1) individualizing foundations (ie, care and fairness), which concern the value of the individual, and (2) binding foundations (ie, loyalty, authority, and sanctity), which concern group integrity [44].

    Individualizing moral foundations (ie, foundations care and harm, and fairness and cheating) are linked to protective COVID-19 behaviors [31,32]. Existing work suggests that individualizing foundations are the most relevant in moral decisions when faced with a disease threat [33]. Additionally, the polarized state of US politics may have contributed to uncivil COVID-19 discussion on social media platforms. Past research has shown that liberal individuals tend to value individualizing foundations, whereas conservative individuals tend to value binding foundations, though some studies have shown that conservatives may value all 5 foundations equally [44-48]. Additionally, scholars have pointed out that individuals who value individualizing foundations are more likely to respond emotionally to uncivil comments [49]. Liberal individuals may experience cognitive dissonance and stress when faced with antivaccine mandate messaging [40,41]. These individuals may be more likely to engage in uncivil behaviors and attack antivaccine mandate messaging promoters to reduce discomfort. Scholars have pointed out that individualizing foundations such as care and fairness are more important to liberals than conservatives, although these foundations are not limited to one political ideology [44]. Given the heavily politicized nature of COVID-19 [5], violating the care and fairness foundations may elicit COVID-19 vaccine discourse incivility. Therefore, we hypothesize that:

    • H2: Care will positively predict COVID-19 vaccine mandate incivility, and
    • H3: Fairness will positively predict COVID-19 vaccine mandate incivility.
    Binding Foundations (Loyalty and Betrayal)

    Loyalty and betrayal or in-group loyalty refers to a person’s allegiance and devotion to their own group, and it may also have a role in COVID-19 discourse incivility. The pandemic gave rise to 2 different groups, individuals who encouraged protective pandemic behaviors (eg, masking and vaccinating) and individuals who disregarded protective measures (eg, 1 study observed that non–mask wearers were more likely to cooperate with other non–mask wearers than mask wearers) [14], suggesting the influence of an in-group bias. Another study observed that COVID-19 discussion on social media platforms was politicized, with right-leaning users tending to engage less with health-promoting hashtags on Twitter (now known as X) than left-leaning users [3]. These findings are in conjunction with existing research, which argues that opinions about the COVID-19 pandemic and vaccine are split along partisan lines, with Republicans exhibiting negative attitudes toward the vaccine [50] and reporting lower intentions to get vaccinated due to increased misperceptions about side effects [51]. In turn, the polarization surrounding COVID-19 could have resulted in partisan-motivated reasoning, where individuals’ prior proattitudinal beliefs and their partisan alignment drive information processing [52]. As such, conservatives may have engaged in uncivil discourse against the COVID-19 vaccine, whereas liberals may have engaged in uncivil discourse in favor of the vaccine to ensure that their vaccine stance aligns with their prior political beliefs. On the basis of the overview of existing research above, we hypothesize that:

    • H4: In-group loyalty will positively predict COVID-19 vaccine mandate incivility.
    Authority and Subversion

    Existing literature suggests that individuals condemn perceived leadership failures in hierarchical organizations [43]. Thus, authority and subversion foundations may incite incivility. Many protective pandemic guidelines have been created, supported, and sometimes enforced by authority figures and leaders (eg, locally elected politicians and the Centers for Disease Control and Prevention [CDC]). For example, right-wing outlets blasted the COVID-19 vaccine and exacerbated the spread of COVID-19 vaccine misinformation [53]. Additionally, Donald Trump, the US president at the time, continuously referred to COVID-19 as a hoax, whereas Democrats condemned politicians who minimized the pandemic’s threat and vaccine efficacy. Such condemnation may have manifested as uncivil comments directed at these government officials and public health authorities. Therefore, we argue that authority and subversion values can prompt uncivil discourse.

    • H5: Authority will positively predict incivility in discussion about COVID-19 vaccine mandates.
    Social Media Culture Norms

    Organizational research suggests that incivility results from interactions between community and situational norms, which are shaped by organizational policies [17]. Cultural norms and platform capabilities can enable or mitigate incivility (ie, Twitter’s hateful conduct policy, Reddit’s moderation rules, and Facebook’s community standards) [54,55]. Social media platforms also respond to elements of incivility differently due to cultural norms, which can impact the frequency of uncivil interactions on the platforms. Indeed, Facebook is known to have less uncivil discourse than Twitter [56], but intolerant comments are more frequently found on Facebook [57]. In the context of our study, different subreddits across Reddit have various norms that are distinct from each other. For example, the subreddit r/lockdownskepticism has the following description of their rules that users have to agree to before joining: “Interdisciplinary examination of lockdowns & other pandemic policies. We acknowledge the threat of COVID-19. We are also concerned about the policies’ impact on our physical and mental health, human rights, and economy. This is a non-partisan, inclusive, global sub. We are empirically minded and do not tolerate unsupported claims or conspiracy theories. **Warning: users may be auto-banned from other subs for posting here**.” In contrast, the subreddit r/coronaviruscirclejerk has a description that starts with, “We are all going to die,” and is more focused on the memes and discussion resulting from the discourse between “panic-filled” individuals and “alarmists” on the web. This indicates some cultural norm differences that exist among subreddits on Reddit. Other social media sites may have different cultural norms that may impact incivility differently. Shmargad et al [58] point out that the frequency of incivility on platforms is dependent on platform norms (eg, moderator rules). On the basis of this discussion, we pose the following RQ:

    • RQ2: Does discourse incivility vary by cultural norms?

    Recruitment

    Data were scraped from the Reddit website, a social media platform where anonymous users may post and interact with content organized into certain communities termed “subreddits.” We examined subreddits centered around COVID-19–related discussions and were likely to have uncivil discourse. For instance, one subreddit of interest was r/nonewnormal, a place for people to discuss and criticize COVID-19 lifestyle disruptions. R/nonewnormal would have been a worthwhile subreddit to scrape data from, but because it was banned for its strong antivaccine and antimask content, we could not collect the data [59]. Because we expected users who frequented r/nonewnormal to move and become active in other subreddits, we leveraged the Subreddit Stats website [60,61]. This website provides statistics on various subreddits and the relationships among them. Two subreddits were identified as having a large user overlap with r/nonewnormal, namely, r/coronaviruscirclejerk and r/lockdownskepticism [62]. The 2 subreddits were selected because of their large overlap with the r/nonewnormal subreddit. The subreddit r/coronaviruscirclejerk primarily included satirical posts ridiculing others who worry about COVID-19, whereas the subreddit r/lockdownskepticism focused on a more empirical discussion that questioned the actual effectiveness of pandemic lockdowns and quarantines. Both subreddits, similar to r/nonewnormal, were illuminating given their focus on COVID-19–related discussion and the high likelihood of them containing uncivil discourse.

    All posts and comments from r/coronaviruscirclejerk and r/lockdownskepticism were collected since their inception in March 2022. The collected data set was then filtered for key terms related to words related to the COVID-19 vaccine (eg, “vaccinate” and “Pfizer”) and key terms related to mandates (eg, “forced” and “mandating”; see Textboxes 1 and 2 for details). The research team took a grounded theory approach to key term selection, including a systematic review of existing literature and popular press for relevant terms (see Figure 1 for details). In addition, the terms were selected based on their salience in the r/coronaviruscirclejerk and r/lockdownskepticism subreddits. A 10% random sample of the comments that contained vaccine and mandate terms was retained for analysis.

    Textbox 1. Keywords for vaccines.

    Vaccine keywords

    dose, johnson, J&J, Jnj, pfizer, moderna, covax, vax, vaccine, vaccinate, Vaccinated, vaccinates, vaccinating, needle, inject, injected, injecting, inoculated, inoculates, inoculating, immunization, immunity, immune, shot, shots, jab, jabbed, jabs, booster, boosted, sputnik, mRNA, comirnaty, spikevax, astrazeneca, covishield, vaxzeveria, janssen, coronavac, epivac, epivaccine, convidicea, unvaxed, unvaxxed, unvaccinated, biontech, az, sinopharm, sinovac, covovax, nuvaxovid

    Textbox 2. Keywords for mandates.

    Mandate keywords

    mandate, mandating, mandated, mandates, force, forcing, forced, forces, require, required, requiring, requires, make, making, made, Makes, coerce, coercing, coerced, coerces, must, need, needing, needed, needs, order, ordering, orders, ordered, necessitate, necessitating, Necessitates, necessitated, demand, demands, demanded, demanding, instruct, instructed, instructing, instructs, command, commanding, commanded, commands, freedom, freedoms, liberties, violate, violating, violated, violates, right, rights

    ‎
    Figure 1. Data collection and filtration process flowchart.

    Statistical Analyses

    Data analysis was a 2-pronged approach. We first examined the presence and nature of incivility in comments and then identified the presence of any discussion of the 5 moral foundations in each comment. Perspective application programming interface (API), a tool that uses machine learning to detect comment incivility, was used to measure 5 dimensions of incivility, namely, toxicity, insults, profanity, threat, and identity attack (see Textbox 3 for definitions and examples). Perspective API is a validated tool trained by human coders on large data sets with millions of comments and has been used in studies across various contexts, such as health and politics [17,20,63,64]. Perspective API assigns each comment a score from 0 to 1 per dimension of incivility based on how closely the comment reflected the specific dimension and how likely it was to impact a conversation. To determine the moral foundations reflected in each comment, we used a prevalidated MFT dictionary [44]. This dictionary tool generated the relative frequency of words associated with each moral foundation as a score from 0 to 1. Similar studies have used prevalidated count dictionaries to measure the aspects of COVID-19 discourse [5,65].

    Textbox 3. Description of the moral foundations.

    Sanctity and degradation

    • Characterized by concerns for physical and spiritual purity, along with chastity [30]

    Care and harm

    • Characterized by motivations to protect and care against suffering [29]

    Fairness and cheating

    • Characterized by concerns against unfairness, cheating, and inequities [29,30]

    Loyalty and betrayal

    • Characterized by devotion and allegiance to a group and loyalty [30]

    Authority and subversion

    • Characterized by obedience, respect, and fulfillment of obligations to hierarchical relationships [30]

    Ethical Considerations

    This study does not include any personally identifiable information and only relies on publicly available data. The institutional review board recognizes that the analysis of publicly available data does not fall under human subject research. As such, ethical review and approval were not required for this study.


    A Spearman correlation analysis was conducted to measure the association between the 5 different dimensions of incivility and the 5 different moral foundations (see Table 1).

    Table 1. Five dimensions of comment incivilitya.
    DimensionPerspective APIb definitionExample comments
    Toxicity“A rude, disrespectful, or unreasonable comment that is likely to make people leave a discussion.”“Don’t they realize that Dee Snider (Twisted Sister) loves vaccine mandates and hates anti vax. Fuck that asshole.”
    Insults“Insulting, inflammatory, or negative comment towards a person or a group of people.”“You’re the one calling for your coworkers to burn in hell for requiring basic safety for one another. Vaccine mandates aren’t ‘communist’, they’ve been around longer than the United States. You’re just a selfish prick with their head up ass.”
    Profanity“Swear words, curse words, or other obscene or profane language.”“Fucking Disneyland isn\'t enforcing masks anymore there is no fucking reason for your college to do so especially since they are requiring the vaccine.”
    Threat“Describes an intention to inflict pain, injury, or violence against an individual or group.”“If they come to your house and try to force vaccine on you or your family members, just take knife, scissors, axe, or something really sharp and... you know what to do. Even if they kill you after that, your life will have more meaning than if you would comply. It’s sad that we have to talk about this, but there we are.”
    Identity attack“Negative or hateful comments targeting someone because of their identity.”“You are wrong. The elders are outside protesting. The youth is having gay orgies in clubs and protesting against capitalism but don’t care about forced vaccinations.”

    aDimensions of incivility were measured via Perspective API [66].

    bAPI: application programming interface.

    A significant positive correlation was observed between purity and each of the 5 dimensions of incivility: identity attack (P<.001; 95% CI 0.07-0.11), insult (P<.001; 95% CI 0.11-0.16), toxicity (P<.001; 95% CI 0.10-0.14), profanity (P<.001; 95% CI 0.07-0.11), and threat (P<.001; 95% CI 0.05-0.09). A significant positive correlation was observed between fairness and the dimensions of identity attack (P<.001; 95% CI 0.08-0.12), insult (P<.001; 95% CI 0.05-0.09), toxicity (P<.001; 95% CI 0.04-0.08), and profanity (P<.001; 95% CI 0.02-0.07). A significant positive correlation was observed between harm and the dimensions of identity attack (P=.003; 95% CI 0.02-0.06), insult (P<.001; 95% CI 0.04-0.08), toxicity (P<.001; 95% CI 0.04-0.08), and threat (P<.001; 95% CI 0.05-0.10). A significant positive correlation was observed only between in-group loyalty and identity attack (P=.009; 95% CI 0.02-0.06). There were no significant correlations between authority and any of the dimensions of incivility (see Table 2). In addition, the descriptive results for incivility by subreddit and moral foundations by subreddit are included below (see Tables 3 and 4).

    Table 2. Spearman correlation results among dimensions of incivility and moral foundations (N=8648).
    Moral foundationIdentity attackInsultToxicityProfanityThreat

    		ρP valueaρP valueaρP valueaρP valueaρP valuea
    Purity0.09<.0010.13<.0010.12<.0010.09<.0010.07<.001
    Fairness0.10<.0010.07<.0010.06<.0010.05<.0010.02.62
    Harm0.05.0030.06<.0010.06<.0010.03.180.08<.001
    In-group loyalty0.04.0090.02>.990.00>.99−0.02>.99−0.02.81
    Authority0.00>.990.01>.990.00>.99−0.02.45−0.01>.99

    aP values were adjusted using the Holm correction.

    Table 3. Summary statistics table for dimensions of incivility by subreddita.
    VariableValue, mean (SD)Value, nValue, SEValue, median (range)
    Identity attack

    		r/lockdownskepticism0.10 (0.11)49610.0020.06 (0.00004-0.94)

    		r/coronaviruscirclejerk0.13 (0.14)36870.0020.08 (0.0007-0.97)
    Threat

    		r/lockdownskepticism0.18 (0.19)49610.0030.10 (0.0007-0.98)

    		r/coronaviruscirclejerk0.23 (0.22)36870.0040.13 (0.006-0.99)
    Insult

    		r/lockdownskepticism0.16 (0.20)49610.0030.08 (0.0004-0.98)

    		r/coronaviruscirclejerk0.20 (0.23)36870.0040.11 (0.002-0.98)
    Toxicity

    		r/lockdownskepticism0.19 (0.21)49610.0030.10 (0.0004-0.99)

    		r/coronaviruscirclejerk0.24 (0.25)36870.0040.13 (0.004-0.99)
    Profanity

    		r/lockdownskepticism0.11 (0.21)49610.0030.04 (0.0002-0.98)

    		r/coronaviruscirclejerk0.15 (0.25)36870.004 0.05 (0.001-0.99)

    aThe Google Perspective application programming interface was used to measure dimensions of incivility.

    Table 4. Summary statistics table for moral foundations by subreddita.
    VariableValue, mean (SD)Value, nValue, SEValue, median (range)
    Harm

    		r/lockdownskepticism0.34 (1.36)49610.020.00 (0.00-15.38)

    		r/coronaviruscirclejerk0.43 (1.67)36870.030.00 (0.00-22.22)
    Fairness

    		r/lockdownskepticism0.23 (0.14)49610.020.00 (0.00-18.18)

    		r/coronaviruscirclejerk0.18 (1.05)36870.020.00 (0.00-16.67)
    Authority

    		r/lockdownskepticism0.37 (1.36)49610.020.00 (0.00-20.00)

    		r/coronaviruscirclejerk0.37 (1.46)36870.020.00 (0.00-16.67)
    In-group

    		r/lockdownskepticism0.13 (0.87)49610.010.00 (0.00-20.00)

    		r/coronaviruscirclejerk0.14 (0.94)36870.020.00 (0.00-18.75)
    Purity

    		r/lockdownskepticism0.13 (0.87)49610.010.00 (0.00-16.67)

    		r/coronaviruscirclejerk0.18 (1.12)36870.020.00 (0.00-16.67)

    aA prevalidated moral foundation dictionary was used to assess the moral foundations of the posts through a computer-assisted text scanner.

    A 2-tailed independent samples z test was conducted to examine whether the mean of identity attack, insult, toxicity, profanity, and threat was different in the r/lockdownskepticism and r/coronaviruscirclejerk subreddits. Results were significant for all 5 dimensions of incivility (P<.001), indicating that the null hypotheses can be rejected (see Table 5 for details). This suggests that the mean of identity attack, insult, toxicity, profanity, and threat in the r/lockdownskepticism subreddit was significantly lower than that in the r/coronaviruscirclejerk subreddit.

    Table 5. Two-tailed independent samples z test for incivility by subreddit (N=8648).
    Variabler/lockdownskepticism, mean (SD)r/coronaviruscirclejerk, mean (SD)zP value
    Toxicity0.19 (0.21)0.24 (0.25)−10.33<.001
    Identity attack0.10 (0.11)0.13 (0.14)−9.32<.001
    Threat0.18 (0.19)0.23 (0.22)−10.55<.001
    Insult0.16 (0.20)0.20 (0.23)−8.82<.001
    Profanity0.11 (0.21)0.15 (0.25)−7.91<.001

    Principal Findings

    Understanding incivility surrounding public health initiatives, such as COVID-19 vaccine mandates, is imperative to improve public health efforts and public well-being. This work revealed differing associations between the moral foundations and each dimension of comment incivility.

    First, as purity increased, so did all 5 dimensions of comment incivility, supporting hypothesis H1 as expected. The following finding is consistent with the study by Frimer et al [39], concluding that purity has been an area of political disagreement. The association between purity and incivility also supports the idea that there may indeed be 2 sides to purity and can explain why purity can predict both COVID-19 protective behaviors and vaccine hesitancy, as seen in recent research [32]. In addition, according to Amin et al [37], current provaccine messaging has been created with a focus on the harm and fairness foundations. However, public health professionals should consider designing messages appealing to other moral foundations, like purity. For example, messages could be designed keeping in mind the domain words associated with the moral foundation (eg, “Getting a vaccine can help your body fight against the impure COVID-19 virus”) [44].

    Second, as fairness increased, all dimensions of comment incivility except for comment threat tended to increase, partially supporting hypothesis H2. Fairness did not predict comment threat, which may be, in part, due to the political polarization associated with moral foundations and COVID-19. In a research report, Harward et al [67] provide an example of how conservative individuals have viewed fairness in society as a “get what you deserve” system instead of a system of equity. It is possible that some conservative individuals may not have felt prompted to make threats against vaccine proponents because they believe proponents have already posed a threat to themselves by getting an “impure” vaccine. This finding implies that COVID-19 vaccine incivility is influenced heavily by the purity foundation, but in this context, it manifests as a fairness concern. This finding increases the need for public health officials to investigate how valuing purity relates to vaccine uptake and, consequently, tailor messaging to address vaccine impurity concerns.

    Third, as harm increased, all dimensions of comment incivility except for comment profanity tended to increase, partially supporting hypothesis H3. One potential explanation for this is a finding from Feldman et al [68] that notes a positive relationship between profanity and honesty. Individuals may have been dishonest when engaging in uncivil discussion involving the care and harm foundation. Additionally, it is possible that although there is some incivility correlated with harm, it may be a facade to hide one’s genuine concerns with COVID-19 vaccine mandates. An implication of this is that public health officials may have been misled by what individuals were concerned about regarding COVID-19 vaccine mandates. This offers one explanation for the prevalence of existing provaccine messaging that appeals to the value of harm [37]. Public health officials should be mindful of the possibility that concerns of fairness surrounding the COVID-19 vaccine mandates may not be as prominent as observed.

    Fourth, as in-group loyalty increased, only comment identity attack tended to increase, partially supporting hypothesis H4. One study found no association between the amount of “exclusionary language” used and comment toxicity [69]. Also, Brewer [70] notes that out-group hostility may also have resulted from a desire to gain political power, a possibility given that COVID-19 has been politicized [3]. This implies that a small part of the uncivil COVID-19 discourse may be due to an issue of in-group loyalty. To address this concern and increase vaccination numbers, public health officials and lawmakers should find a way to bridge partisan gaps in the United States.

    Fifth, there was no correlation between authority and any of the dimensions of incivility, refuting hypothesis H5. One possible reason is that discussion about COVID-19 vaccine mandates may be more of a political in-group concern rather than a concern of authority, given that health information has been heavily politicized [3,71]. Although contrary to our hypothesis, this result still has important implications. For instance, these findings suggest that uncivil discourse regarding COVID-19 vaccine mandates has been an issue of policy and preservation rather than an issue of the policy makers and policy endorsers. In other words, the incivility surrounding COVID-19 vaccine mandates has little to do with authority figures such as the CDC, the president, or local public health authorities. This suggests that public health efforts may not have been affected by the reputation of authority figures but that they are affected by the way public health policies are perceived.

    Further, we investigated whether incivility varies by platform norms. The findings revealed that the means of the different dimensions of incivility (eg, toxicity, severe toxicity, insult, profanity, threat, and identity attacks) in the r/lockdownskepticism subreddit were significantly lower than those in the r/coronaviruscirclejerk subreddit, suggesting that incivility does vary by platform norms. Recent studies have argued that platform norms vary within web-based cultures and within the microcultures of these platforms [17]. Because Reddit moderator roles differ for each subreddit, our results point toward variations within Reddit in discourse incivility. Simply put, discourse incivility can vary within Reddit as moderators have different rules that can impede or foster incivility. These results are consistent with prior research that has pointed to variations in incivility by platform [56,57,72].

    Practical Implications

    The COVID-19 pandemic has caused disruptions in all facets of life worldwide. Therefore, restoring trust in public health agencies and protocols is paramount. Tailored public health messaging incorporating social values (eg, moral foundations) may help to reach, educate, and persuade individuals in a way that evokes civil responses and improve compliance [73-76]. One way health messages can be tailored is by addressing the moral values underlying the concerns people may have about public health interventions (eg, vaccinations and screenings), which could manifest as uncivil discourse. Existing literature has argued that developing customized messaging based on moral foundations can be effective in persuading individuals. Specifically, scholars have found that reframing messages based on moral foundations congruent to individual attitudes can persuade conservatives and liberals to agree on environmental issues [77] and enhance participation in sustainable environmental practices [78]. Other studies have found that issues framed using moral foundations can not only strengthen existing attitudes but also shift attitudes among liberals and conservatives [79]. Another approach can be to customize public health messages based on the levels of incivility observed in different areas of social media platforms. For instance, in our study, we found that the r/coronaviruscirclejerk subreddit had higher levels of different dimensions of incivility than r/lockdownskepticism. There is also evidence that Twitter is less toxic than Facebook as more uncivil comments are found on Facebook [56,57]. Given these platform-based cultural differences, it may be beneficial for health care professionals and lawmakers to develop messages targeted at these differences. Specifically, these messages could be directed toward areas within a platform (ie, Reddit and Facebook) where incivility is more salient. Ultimately, we argue that the antecedents of incivility can inform public health interventions. Existing studies have found a link between incivility and negative emotions such as sadness and anxiety [20]. Therefore, public health messaging that evokes such emotions could result in uncivil discourse from the public. However, the underlying mechanisms fueling these emotional reactions remain unclear. By considering the moral foundations fueling incivility, public health officials can design effective messages aimed at appealing to specific moral foundations that will, in turn, increase vaccine uptake and overall community well-being.

    Limitations

    This study is novel in that it uses a data set comprising posts from users who are banned from the antivaccine subreddits, which sheds light on unique and otherwise unexplored discourse. Another strength of our research is that it focuses on Reddit, a platform that is often overshadowed by Facebook and Twitter in extant literature. However, our study is also limited in that it focused on 2 specific subreddits, which are not representative of all vaccine-related discourse on Reddit and other social media. Future studies may benefit by focusing on comparing vaccine-related discourse across different social media platforms with a more comprehensive data set. Another limitation of our research is its observational nature that prevents us from implying causation. An extension of this study could focus on establishing a causal relationship between moral foundations and COVID-19 vaccine mandate incivility, if such a relationship exists. Also, our study used only 10% of the sample because of scarce computational resources, which limited us in our analyses. Future researchers should also consider investigating how moral foundations interact with different dimensions of incivility with a larger data set. In addition, future research can also use qualitative approaches and interview users who post uncivil conduct on social media sites such as Reddit. Because incivility is a multifaceted construct, such analyses can aid our understanding of incivility and shed light on the psychological processes that lead to uncivil discourse.

    Conclusions

    This paper examined whether moral foundations were related to incivility surrounding COVID-19 vaccine discourse. We found that purity, fairness, harm, and in-group loyalty were positively related to different dimensions of incivility. This study adds to the growing literature focused on theorizing the mechanisms behind incivility related to COVID-19–related discourse. Our findings highlight the need for health campaigns to design messages appealing to specific moral foundations of specific demographics. For example, organizations such as the CDC have already created messaging that highlights moral foundations such as care and fairness by highlighting the protective nature of vaccines and their availability for all individuals (see Figure 2 for details). By integrating moral foundations, messaging related to COVID-19 may be an effective way to persuade audiences to follow public health protocols and engage in civil discourse.

    ‎
    Figure 2. Centers for Disease Control and Prevention (CDC) messaging highlighting the care and fairness moral foundations.

    Data Availability

    The data sets generated during and/or analyzed during this study are available from the corresponding author on reasonable request.

    Conflicts of Interest

    None declared.

    1. WHO coronavirus (COVID-19) dashboard. World Health Organization. 2022. URL: https://covid19.who.int/ [accessed 2022-07-30]
    2. COVID data tracker. Centers for Disease Control and Prevention. 2022. URL: https://covid.cdc.gov/covid-data-tracker/#datatracker-home [accessed 2022-07-30]
    3. Jiang J, Chen E, Yan S, Lerman K, Ferrara E. Political polarization drives online conversations about COVID-19 in the United States. Hum Behav Emerg Technol. 2020;2(3):200-211. [FREE Full text] [CrossRef] [Medline]
    4. Sabahelzain MM, Hartigan-Go K, Larson HJ. The politics of Covid-19 vaccine confidence. Curr Opin Immunol. 2021;71:92-96. [FREE Full text] [CrossRef] [Medline]
    5. Stevens H, Palomares NA. Constituents' inferences of local governments' goals and the relationship between political party and belief in COVID-19 misinformation: cross-sectional survey of Twitter followers of State Public Health Departments. JMIR Infodemiology. 2022;2(1):e29246. [FREE Full text] [CrossRef] [Medline]
    6. Kennedy J. Populist politics and vaccine hesitancy in Western Europe: an analysis of national-level data. Eur J Public Health. 2019;29(3):512-516. [FREE Full text] [CrossRef] [Medline]
    7. Largent EA, Persad G, Sangenito S, Glickman A, Boyle C, Emanuel EJ. US public attitudes toward COVID-19 vaccine mandates. JAMA Netw Open. 2020;3(12):e2033324. [FREE Full text] [CrossRef] [Medline]
    8. Zhao N, Zhou G. COVID-19 stress and addictive Social Media Use (SMU): mediating role of active use and social media flow. Front Psychiatry. 2021;12:635546. [FREE Full text] [CrossRef] [Medline]
    9. Cinelli M, Quattrociocchi W, Galeazzi A, Valensise CM, Brugnoli E, Schmidt AL, et al. The COVID-19 social media infodemic. Sci Rep. 2020;10(1):16598. [FREE Full text] [CrossRef]
    10. Lovari A. Spreading (dis)trust: COVID-19 misinformation and government intervention in Italy. Media Commun. 2020;8(2):458-461. [FREE Full text] [CrossRef]
    11. Ahmed S, Rasul ME. Social media news use and COVID-19 misinformation engagement: survey study. J Med Internet Res. 2022;24(9):e38944. [FREE Full text] [CrossRef] [Medline]
    12. Ahmed S, Rasul ME. Examining the association between social media fatigue, cognitive ability, narcissism and misinformation sharing: cross-national evidence from eight countries. Sci Rep. 2023;13(1):15416. [FREE Full text] [CrossRef] [Medline]
    13. Bridgman A, Merkley E, Loewen PJ, Owen T, Ruths D, Teichmann L, et al. The causes and consequences of COVID-19 misperceptions: understanding the role of news and social media. Harvard Kennedy School (HKS) Misinformation Review. Jun 18, 2020 [FREE Full text] [CrossRef]
    14. Powdthavee N, Riyanto YE, Wong ECL, Yeo JXW, Chan QY. When face masks signal social identity: explaining the deep face-mask divide during the COVID-19 pandemic. PLoS One. 2021;16(6):e0253195. [FREE Full text] [CrossRef] [Medline]
    15. Chen GM, Muddiman A, Wilner T, Pariser E, Stroud NJ. We should not get rid of incivility online. Soc Media Soc. 2019;5(3):205630511986264. [FREE Full text] [CrossRef]
    16. Muddiman A, Stroud NJ. News values, cognitive biases, and partisan incivility in comment sections. J Commun. 2017;67(4):586-609. [CrossRef]
    17. Stevens H, Acic I, Taylor LD. Uncivil reactions to sexual assault online: linguistic features of news reports predict discourse incivility. Cyberpsychol Behav Soc Netw. 2021;24(12):815-821. [CrossRef] [Medline]
    18. Sun Q, Wojcieszak M, Davidson S. Over-time trends in incivility on social media: evidence from political, non-political, and mixed sub-reddits over eleven years. Front Polit Sci. 2021;3:741605. [FREE Full text] [CrossRef]
    19. Theocharis Y, Barberá P, Fazekas Z, Popa SA. The dynamics of political incivility on Twitter. SAGE Open. 2020;10(2):215824402091944. [FREE Full text] [CrossRef]
    20. Stevens H, Rasul ME, Oh YJ. Emotions and incivility in vaccine mandate discourse: natural language processing insights. JMIR Infodemiology. 2022;2(2):e37635. [FREE Full text] [CrossRef] [Medline]
    21. Rasul ME, Ahmed S. Not all conservatives are vaccine hesitant: examining the influence of misinformation exposure, political ideology, and flu vaccine acceptance on COVID-19 vaccine hesitancy. Vaccines (Basel). 2023;11(3):586. [FREE Full text] [CrossRef] [Medline]
    22. Coe K, Kenski K, Rains SA. Online and uncivil? patterns and determinants of incivility in newspaper website comments. J Commun. 2014;64(4):658-679. [CrossRef]
    23. Gervais BT. Incivility online: affective and behavioral reactions to uncivil political posts in a web-based experiment. J Inf Technol Politics. 2015;12(2):167-185. [CrossRef]
    24. Sobieraj S, Berry JM. From incivility to outrage: political discourse in blogs, talk radio, and cable news. Polit Commun. 2011;28(1):19-41. [CrossRef]
    25. Stroud NJ, Scacco JM, Muddiman A, Curry AL. Changing deliberative norms on news organizations' Facebook sites. J Comput Mediat Commun. 2014;20(2):188-203. [FREE Full text] [CrossRef]
    26. Anderson AA, Brossard D, Scheufele DA, Xenos MA, Ladwig P. The “nasty effect”: online incivility and risk perceptions of emerging technologies. J Comput Mediat Commun. 2013;19(3):373-387. [FREE Full text] [CrossRef]
    27. Caza BB, Cortina LM. From insult to injury: explaining the impact of incivility. Basic Appl Soc Psych. 2007;29(4):335-350. [CrossRef]
    28. Maity SK, Chakraborty A, Goyal P, Mukherjee A. Opinion conflicts: an effective route to detect incivility in Twitter. Proc ACM Hum Comput Interact. 2018;2(CSCW):1-27. [CrossRef]
    29. Graham J, Haidt J, Koleva S, Motyl M, Iyer R, Wojcik S, et al. Moral foundations theory. In: Olson JM, Zanna MP, editors. Advances in Experimental Social Psychology. San Diego, CA, USA. Elsevier Science; 2013;55-130.
    30. Haidt J, Graham J, Joseph C. Above and below left–right: ideological narratives and moral foundations. Psychol Inq. 2009;20(2-3):110-119. [CrossRef]
    31. Ansani A, Marini M, Cecconi C, Dragoni D, Rinallo E, Poggi I, et al. Analyzing the perceived utility of COVID-19 countermeasures: the role of pronominalization, moral foundations, moral disengagement, fake news embracing, and health anxiety. Psychol Rep. 2022;125(5):2591-2622. [FREE Full text] [CrossRef] [Medline]
    32. Chan EY. Moral foundations underlying behavioral compliance during the COVID-19 pandemic. Pers Individ Dif. 2021;171:110463. [FREE Full text] [CrossRef] [Medline]
    33. Ekici H, Yücel E, Cesur S. Deciding between moral priorities and COVID-19 avoiding behaviors: a moral foundations vignette study. Curr Psychol. 2023;42(7):5922-5938. [FREE Full text] [CrossRef] [Medline]
    34. Kim B, Cooks E, Kim SK. Exploring incivility and moral foundations toward Asians in English-speaking tweets in hate crime-reporting cities during the COVID-19 pandemic. Internet Res. 2021;32(1):362-378. [CrossRef]
    35. About Reddit. Reddit. 2015. URL: https://www.reddit.com/about [accessed 2022-08-17]
    36. Shatz I. Fast, free, and targeted: Reddit as a source for recruiting participants online. Soc Sci Comput Rev. 2016;35(4):537-549. [CrossRef]
    37. Amin AB, Bednarczyk RA, Ray CE, Melchiori KJ, Graham J, Huntsinger JR, et al. Association of moral values with vaccine hesitancy. Nat Hum Behav. 2017;1(12):873-880. [CrossRef] [Medline]
    38. Vartanova I, Eriksson K, Hazin I, Strimling P. Different populations agree on which moral arguments underlie which opinions. Front Psychol. 2021;12:648405. [FREE Full text] [CrossRef] [Medline]
    39. Frimer JA, Biesanz JC, Walker LJ, MacKinlay CW. Liberals and conservatives rely on common moral foundations when making moral judgments about influential people. J Pers Soc Psychol. 2013;104(6):1040-1059. [CrossRef] [Medline]
    40. Festinger L. A Theory of Cognitive Dissonance, Volume 10. Stanford. Stanford University Press; 1957.
    41. Harmon-Jones E, Mills J. An introduction to cognitive dissonance theory and an overview of current perspectives on the theory. In: Harmon-Jones E, editor. Cognitive Dissonance: Reexamining a Pivotal Theory in Psychology, 2nd Edition. Washington, DC. American Psychological Association; 2019;3-24.
    42. Armstrong GM, Wronski J. Framing hate: moral foundations, party cues, and (in)tolerance of offensive speech. J Soc Polit Psychol. 2019;7(2):695-725. [FREE Full text] [CrossRef]
    43. Haidt J, Graham J. When morality opposes justice: conservatives have moral intuitions that liberals may not recognize. Soc Just Res. 2007;20(1):98-116. [CrossRef]
    44. Graham J, Haidt J, Nosek BA. Liberals and conservatives rely on different sets of moral foundations. J Pers Soc Psychol. 2009;96(5):1029-1046. [CrossRef] [Medline]
    45. Graham J, Nosek BA, Haidt J. The moral stereotypes of liberals and conservatives: exaggeration of differences across the political spectrum. PLoS One. 2012;7(12):e50092. [FREE Full text] [CrossRef] [Medline]
    46. McAdams DP, Albaugh M, Farber E, Daniels J, Logan RL, Olson B. Family metaphors and moral intuitions: how conservatives and liberals narrate their lives. J Pers Soc Psychol. 2008;95(4):978-990. [CrossRef] [Medline]
    47. Turner-Zwinkels FM, Johnson BB, Sibley CG, Brandt MJ. Conservatives' moral foundations are more densely connected than liberals' moral foundations. Pers Soc Psychol Bull. 2021;47(2):167-184. [FREE Full text] [CrossRef] [Medline]
    48. Voelkel JG, Brandt MJ. The effect of ideological identification on the endorsement of moral values depends on the target group. Pers Soc Psychol Bull. 2019;45(6):851-863. [FREE Full text] [CrossRef] [Medline]
    49. Walter A, Lipsitz K. How moral value commitments shape responses to political civility and incivility. Am Polit Res. 2021;49(4):359-367. [CrossRef]
    50. Fridman A, Gershon R, Gneezy A. COVID-19 and vaccine hesitancy: a longitudinal study. PLoS One. 2021;16(4):e0250123. [FREE Full text] [CrossRef] [Medline]
    51. Kreps SE, Goldfarb JL, Brownstein JS, Kriner DL. The relationship between US adults' misconceptions about COVID-19 vaccines and vaccination preferences. Vaccines (Basel). 2021;9(8):901. [FREE Full text] [CrossRef] [Medline]
    52. Bolsen T, Druckman JN, Cook FL. The influence of partisan motivated reasoning on public opinion. Polit Behav. 2013;36(2):235-262. [CrossRef]
    53. Motta M, Stecula D, Farhart C. How right-leaning media coverage of COVID-19 facilitated the spread of misinformation in the early stages of the pandemic in the U.S. Can J Pol Sci. 2020;53(2):335-342. [FREE Full text] [CrossRef]
    54. Jhaver S, Birman I, Gilbert E, Bruckman A. Human-machine collaboration for content regulation: the case of Reddit automoderator. ACM Trans Comput Hum Interact. 2019;26(5):1-35. [FREE Full text] [CrossRef]
    55. Twitter's policy on hateful conduct. Twitter Help Center. 2022. URL: https://help.twitter.com/en/rules-and-policies/hateful-conduct-policy [accessed 2022-05-31]
    56. Oz M, Zheng P, Chen GM. Twitter versus Facebook: comparing incivility, impoliteness, and deliberative attributes. New Media Soc. 2017;20(9):3400-3419. [CrossRef]
    57. Rossini P. Beyond incivility: understanding patterns of uncivil and intolerant discourse in online political talk. Communic Res. 2020;49(3):399-425. [CrossRef]
    58. Shmargad Y, Coe K, Kenski K, Rains SA. Social norms and the dynamics of online incivility. Soc Sci Comput Rev. 2021;40(3):717-735. [CrossRef]
    59. Schiltz H. Reddit banned an anti-vaccine, anti-mask community after 135 of its biggest forums protested. Business Insider. URL: https:/​/www.​businessinsider.com/​reddit-bans-nonewnormal-subreddit-that-spread-covid-19-vaccine- misinformation-2021-9 [accessed 2023-11-02]
    60. We are all going to die. Subredditstats.com. URL: https://subredditstats.com/r/coronaviruscirclejerk [accessed 2023-11-25]
    61. LockdownSkepticism. Subredditstats.com. URL: https://subredditstats.com/r/lockdownskepticism [accessed 2023-11-25]
    62. Related subreddits to r/coronaviruscirclejerk by user/redditor overlap. Reddit. URL: https://subredditstats.com/subreddit-user-overlaps/nonewnormal [accessed 2022-02-24]
    63. Hopp T, Vargo CJ, Dixon L, Thain N. Correlating self-report and trace data measures of incivility: a proof of concept. Soc Sci Comput Rev. 2018;38(5):584-599. [CrossRef]
    64. Pascual-Ferrá P, Alperstein N, Barnett DJ. A multi-platform approach to monitoring negative dominance for COVID-19 vaccine-related information online. Disaster Med Public Health Prep. 2021:1-24. [FREE Full text] [CrossRef] [Medline]
    65. Stevens HR, Acic I, Rhea S. Natural language processing insight into LGBTQ+ youth mental health during the COVID-19 pandemic: longitudinal content analysis of anxiety-provoking topics and trends in emotion in LGBTeens microcommunity subreddit. JMIR Public Health Surveill. 2021;7(8):e29029. [FREE Full text] [CrossRef] [Medline]
    66. Perspective. URL: https://developers.perspectiveapi.com/s/about-the-api-attributes-and-languages?language=en_US [accessed 2023-11-25]
    67. Harward B, Taylor A, Kavanagh S. What's fair? exploring the behavioral science of justice and fairness (part 2). Government Finance Officers Association. 2021. URL: https://www.gfoa.org/materials/whats-fair-2 [accessed 2023-11-02]
    68. Feldman G, Lian H, Kosinski M, Stillwell D. Frankly, we do give a damn: the relationship between profanity and honesty. Soc Psychol Personal Sci. 2017;8(7):816-826. [FREE Full text] [CrossRef] [Medline]
    69. Hansen RW. You’ve never been welcome here: exploring the relationship between exclusivity and incivility in online forums. J Inf Technol Politics. 2022;20(2):139-153. [CrossRef]
    70. Brewer MB. The psychology of prejudice: ingroup love and outgroup hate? J Soc Issues. 2002;55(3):429-444. [CrossRef]
    71. Haidt J, Joseph C. Intuitive ethics: how innately prepared intuitions generate culturally variable virtues. Daedalus. 2004;133(4):55-66. [FREE Full text] [CrossRef]
    72. Szabó G, Kmetty Z, Molnár E. Politics and incivility in the online comments: what is beyond the norm-violation approach? Int J Comm. 2021;15:1659-1684. [FREE Full text]
    73. Bull FC, Kreuter MW, Scharff DP. Effects of tailored, personalized and general health messages on physical activity. Patient Educ Couns. 1999;36(2):181-192. [CrossRef] [Medline]
    74. Jennings FJ, Russell FM. Civility, credibility, and health information: the impact of uncivil comments and source credibility on attitudes about vaccines. Public Underst Sci. 2019;28(4):417-432. [CrossRef] [Medline]
    75. Reyna VF. A scientific theory of gist communication and misinformation resistance, with implications for health, education, and policy. Proc Natl Acad Sci U S A. 2021;118(15):e1912441117. [FREE Full text] [CrossRef] [Medline]
    76. Yom-Tov E, Shembekar J, Barclay S, Muennig P. The effectiveness of public health advertisements to promote health: a randomized-controlled trial on 794,000 participants. NPJ Digit Med. 2018;1:24. [FREE Full text] [CrossRef] [Medline]
    77. Feinberg M, Willer R. The moral roots of environmental attitudes. Psychol Sci. 2013;24(1):56-62. [CrossRef] [Medline]
    78. Kidwell B, Farmer A, Hardesty DM. Getting liberals and conservatives to go green: political ideology and congruent appeals. J Consum Res. 2013;40(2):350-367. [CrossRef]
    79. Day MV, Fiske ST, Downing EL, Trail TE. Shifting liberal and conservative attitudes using moral foundations theory. Pers Soc Psychol Bull. 2014;40(12):1559-1573. [FREE Full text] [CrossRef] [Medline]

    ‎
    API: application programming interface
    CDC: Centers for Disease Control and Prevention
    LGBTQ: lesbian, gay, bisexual, transgender, queer
    MFT: moral foundations theory
    RQ: research question

    Edited by A Mavragani; submitted 28.06.23; peer-reviewed by YJ Oh, A Koskan; comments to author 13.09.23; revised version received 11.10.23; accepted 29.10.23; published 29.11.23.

    Copyright

    ©Jason Tin, Hannah Stevens, Muhammad Ehab Rasul, Laramie D Taylor. Originally published in JMIR Formative Research (https://formative.jmir.org), 29.11.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.