Crowdsourcing Adverse Events Associated With Monoclonal Antibodies Targeting Calcitonin Gene–Related Peptide Signaling for Migraine Prevention: Natural Language Processing Analysis of Social Media

Introduction

Preventive medications for migraine include both non–disease-specific drugs (eg, antihypertensives and anticonvulsants) and, more recently, disease-specific anti–calcitonin gene–related peptide (CGRP) therapies [1]. Randomized controlled trials (RCTs) demonstrate the clinically relevant efficacy and tolerability of anti-CGRP monoclonal antibodies (mAbs) [2]. However, they may not accurately reflect the real-world experiences of more heterogeneous and diverse patient populations who often have a higher disease burden and more comorbidities [3-6]. In addition to direct reports of potential adverse events from health care providers and patients, regulatory organizations encourage marketing authorization holders to screen digital media for suspected adverse reactions, which should be handled similarly to spontaneous reports with the same requirements applied [7]. While social media represents an obvious venue in this context, its usage for pharmacovigilance is still in its infancy [8]. Complementary methods of postmarketing surveillance for adverse events with social media are needed.

The last 15 years were characterized by a vast proliferation of social media platforms, allowing for both the rapid dissemination of medical information, as well as direct insight into patient perspectives. Indeed, migraine is most prevalent in individuals between 15-49 years of age, who are among the most active users of the internet [9]. This is also reflected by the growth of migraine-related content on the internet [10-12]. The anonymous social media discussion board Reddit has over 430 million monthly active users and serves as a popular forum for health-related discussions [13]. Unlike other digital media platforms, Reddit provides easily accessible and legal ways of accessing historic forum data. Anonymous forums like Reddit allow users to share personal medical experiences and advice without social stigma. Collectively, the migraine-related subforums (or “subreddits”) such as r/Migraine have a user base exceeding 54,100. While these platforms allow for a democratic exploration of patient experiences and perspectives, they are relatively understudied.

Computational linguistics, a discipline encompassing automatic manipulation and quantitative analysis of oral or written language, represents a potential way to explore this content. Computational linguistics can be applied to identify clusters of words in a dataset with the goal of deriving dominant thematic elements. Identifying word distribution frequencies in specific groups of texts is a common technique used in computational linguistics with multiple wide-ranging applications [14]. In this study, we applied computational linguistics, specifically word frequency analysis, to characterize and identify self-reported adverse events associated with the new therapeutic anti-CGRP mAbs reported by users on Reddit. To ascertain the reliability and validity of our methodology, we first conducted a preliminary study with older-generation migraine drugs to see if we could identify known adverse reactions. After validation, we applied the methodology to the newer generation of CGRP therapies for migraine prevention.

Methods

Study Design

This study was a cross-sectional survey of posts on the r/Migraine subforum on Reddit. Data were analyzed using custom back-end software. The study consisted of four phases: (1) data extraction, (2) automated data organization, (3) validation of computational linguistics approach, and (4) application of computational linguistics approach.

Data Extraction

We extracted posts in JavaScript Object Notation format from the Reddit r/Migraine subforum published from January 1, 2010, to January 1, 2020, using Pushshift, a publicly available Reddit repository, in order to extract data for analysis [15]. The time stamp, title, and text of each post were obtained and combined into a comma-separated values file for further evaluation. Of note, our search did not include any comments responding to individual posts. Furthermore, usernames were excluded to ensure the protection of personal data. Once the above data were downloaded, we then converted the entire document to lowercase characters for ease of data search and extraction.

Automated Data Organization

For the validation phase of the study, we algorithmically obtained, through the Haskell filter function, a list of documents containing posts with either the brand or trade name for propranolol and topiramate. These medications were selected as evidence-based preventive treatments for migraine in accordance with American Headache Society guidelines and because their side effect profiles are well known [16]. Next, we algorithmically compiled all posts containing each drug in separate documents. To use computational linguistics terminology, each document represents a corpus for a medication; each post within that corpus further represents a text.

We subsequently identified posts not including any of the above medications, nor including any CGRP therapeutic medication, to serve as controls for comparison. To conduct multiple tests, we randomly divided these controls into 4 groups: group 1, group 2, group 3, and group 4. To accomplish this, we assigned each post in the control a random number between 1 to 4. All posts with randomly assigned digits 1 were grouped into group 1. Those with a digit of 2 were grouped into group 2, and so on.

Validation of Computational Linguistics Approach

Comparing the relative occurrence of words between 2 groups of texts is a well-defined technique in computational linguistics. For drugs with different distinct side effect profiles (eg, the anticonvulsant topiramate vs the beta-blocker propranolol), we hypothesized there would be a difference in the frequency of word distributions between corpora for these different medications.

Although the chi-square test or log-likelihood ratio test are commonly used, Lijiffijit et al [17] recently demonstrated that the Wilcoxon rank-sum test, Welch t test, or bootstrap test are better choices to use to avoid overconfidence bias. Consequently, we used and validated Welch t test for each of these comparisons given the ease of implementation and intuitiveness.

In accordance with the abovementioned methodology, given a corpus for drug S and a corpus for drug T, we denoted each individual word by q, the total number of words in each corpus by n₁ and n₂, and x₁ and x₂ as the mean frequency of q over corpus S and corpus T, respectively. We then applied Welch t test [17]. To simplify our calculations, words with fewer than 3 characters in length, symbols, and numbers were filtered out. We identified any words showing statistical significance at P<.05 using a 1-tailed t test since to reject the null hypothesis, it is sufficient to show that a specific word occurrence is greater—not less—in 1 corpus against another, or a t statistic>1.73. Because this process involved the repeated application of Welch t test n₁+ n₂ times to the same dataset, the Benjamini-Hochberg procedure was used to decrease multisampling bias. Specifically, once statistically significant words were obtained after a given comparison between 2 corpora, the list of significant words was then ranked by P value. We then applied the Benjamini-Hochberg procedure to this ranked ordered list, only retaining those words passing this procedure.

To validate our method and the reliability of the Reddit data, we applied this approach first by comparing propranolol and topiramate. Of note, if a text existed in both the propranolol and the topiramate corpora, then it was taken out of both. Subsequently, we compared propranolol versus group 1 and topiramate versus group 2.

Application of Computational Linguistics Approach

The above approach was then applied to the corpus between erenumab versus group 3 in order to establish a baseline side effect profile for erenumab. Similarly, fremanezumab was compared to group 4 to establish a baseline side effect profile for fremanezumab. We then compared the erenumab corpus against the fremanezumab corpus to ascertain any difference between the 2 medication side effect profiles. This is clinically relevant, as erenumab targets the CGRP receptor, while fremanezumab targets the CGRP ligand. We were interested to see whether there were fewer reports of specific adverse events with erenumab than with fremanezumab. Again, when comparing erenumab and fremanezumab, shared posts were removed.

For evaluation of our data, all contributing authors reviewed words identified through the above processes, where a word is counted as describing a potential adverse event by consensus. The above algorithms, including data download, parsing, filtering, and data analysis were implemented through custom back-end software using a combination of the Clojure and Haskell programming languages.

Ethical Considerations

Our data source was obtained from Pushshift, a social media data collection site, with the explicit purpose of collecting Reddit data for the purpose of research [15]. Our project proposal was approved by the Rutgers University institutional review board as nonhuman subjects research (Pro2020001648). To ensure the anonymity of users in our data processing, we removed all usernames from our dataset. Furthermore, since our methodology examines posts in aggregate, individual identifiable data were not published nor evaluated as part of this study.

Results

Dataset

We extracted a total of 22,467 Reddit posts from r/Migraine, spanning from January 1, 2010, to December 31, 2019. In the validation phase of the study, we identified 402 (2%) posts that mentioned propranolol and 1423 (6.33%) posts that mentioned topiramate. During the application phase, focusing on mAbs targeting CGRP signaling, we found 468 (2.08%) posts that mentioned erenumab and 73 (0.32%) posts that mentioned fremanezumab (Figure 1).

Since these posts did not exclusively reference the target medications, additional steps were taken to exclude co-occurring texts when comparing medications, as detailed in the Methods section. For the control corpora, group 1 contained 5058 posts, group 2 had 4996 posts, group 3 included 5130 posts, and group 4 comprised 5009 posts.

‎

Validation of Computational Linguistics Approach

When topiramate posts were compared against propranolol posts using the earlier methodology, side effect–related words including “appetite,” “cognitive,” “tingling,” “remember,” “taste,” “dopamax” (a frequently used slang term describing the cognitive adverse events of topiramate), “tastes,” and “kidney” were statistically more frequent (Textbox 1).

When propranolol was compared to the control corpus group 1, a total of 98 words appeared more frequently, none of which we identified as potential adverse events (Textbox 2). When topiramate was compared against group 2, there were 563 words that appeared more often (Textbox 3). The following words were determined to be potential adverse effects: “appetite,” “weight,” “remember,” “memory,” “cognitive,” “anxiety,” “taste,” “zombie,” “tingly,” “dizzy,” “fatigue,” “forget,” “kidney,” “foggy,” “dopamax,” “forgetful,” “dizziness,” “carbonate,” and “tastes.”

Application of Computational Linguistics Approach

We next identified frequent words of interest by comparing erenumab to group 3 (Textbox 4) and fremanezumab to group 4. For erenumab, 36 words appeared more frequently, but “constipation” was the only potential adverse effect. The only word appearing more often for fremanezumab versus the control corpus was “Ajovy,” a brand name for fremanezumab.

When comparing erenumab to fremanezumab, there were 180 words that were significantly associated with erenumab, with the following words representing potential adverse events: “depression,” “constipation,” and “vomiting” (Textbox 5). Notably, the word “muscle” was also associated with erenumab, but it was unclear whether this described an adverse event (Textbox 5). The only word considered significant for fremanezumab was “Ajovy” with no adverse effect–related words detected.

Discussion

In this study, we applied computational linguistics methods to unfiltered discussions about migraine treatments on Reddit to distinguish patient-reported adverse effects associated with these medications. The validation phase of our study accurately identified common adverse events for standard migraine preventive treatments. For example, typical adverse events such as “taste” and “tingling” appeared in posts for topiramate. After applying this methodology to mAbs targeting CGRP or its receptor—fremanezumab and erenumab, respectively—no definite adverse events were flagged for fremanezumab. Notable flagged words for erenumab included “constipation,” “depression,” and “vomiting.”

In our proof-of-concept analysis, we confirmed that keywords discussed in relation to topiramate from the Reddit r/Migraine subforum are consistent with the most common adverse events observed in RCTs [18]. Interestingly, dizziness, a common side effect, did not appear when comparing propranolol to either the control corpus or topiramate [19]. A possible explanation could be that many different words were used to describe this side effect, for example, “lightheadedness” and “dizzy,” which were detected individually but did not meet statistical significance after correcting for multiple comparisons. Because patients may use a variety of different words to describe a single symptom, pooling these terms be useful in future studies when analyzing social media data.

While constipation was not a major concern during RCTs with erenumab, emerging real-world data report that it is among the most frequent reasons for treatment discontinuation [20,21]. Constipation was subsequently added to the US Food and Drug Administration (FDA) label after regulatory approval [22]. During RCTs, approximately 2% of participants reported constipation compared to approximately 10% to 20% of real-world users [3,20,23]. Our findings further support this, since Reddit users frequently discuss constipation and vomiting in relation to erenumab. The discrepancy between data from RCTs and real-world experiences may be due to a more heterogeneous patient population receiving erenumab in the postmarketing phase. This population likely includes patients with other medical comorbidities or factors that could have excluded them from consideration in clinical trials. Increasing evidence suggests that individuals with migraine may have a higher prevalence of comorbid gastrointestinal disorders when compared with the general population [24,25].

Interestingly, the word “muscle” was associated with posts discussing erenumab when compared with fremanezumab. Although we did not include this word as a potential adverse event due to its ambiguity, it could be related to muscle cramps, which were added to the FDA warning label for erenumab after approval [22]. Muscle spasms occurred in approximately 2% of patients in RCTs [20]. Whether “muscle”-related adverse effects are occurring at a more significant rate in real-world use is unclear. Additional investigation into the extent and severity of muscle disease due to erenumab could be considered in future studies or when examining other postmarketing surveillance data sources for verification.

Another notable postmarketing safety update was the addition of hypertension to the FDA label for erenumab; however, this was not flagged in our analysis. In RCTs, less than 1% of participants reported hypertension, but participants with severe cardiovascular risk factors were excluded from these trials, and hypertension has not been commonly noted in other studies [3,23]. It is possible that cardiovascular risk factors are screened to a higher degree than gastrointestinal disorders. Patients at risk for hypertensive complications might, therefore, be more likely to decline treatment with this drug. Finally, worsening of hypertension is typically asymptomatic, and patients may not recognize this as a potential adverse event to raise in social media discussions.

There were no keywords flagged in the fremanezumab dataset. The finding that constipation was flagged for erenumab but not for fremanezumab could be related to pharmacodynamic differences in the mechanism of action. Erenumab targets the CGRP receptor, while fremanezumab targets its ligand. One study offers a possible mechanistic model that strengthens this hypothesis, specifically, via direct effects of CGRP on intestinal motility modulated by other receptors in addition to the canonical CGRP receptor [26]. However, galcanezumab, another mAb targeting the CGRP ligand, also had constipation listed as a common adverse reaction in clinical studies and postmarketing reports [27]. Further data are needed to clarify any potential differences in mechanism.

A strength of our study is the inherent anonymity of the Reddit forum, which allows us to analyze information that patients might not otherwise feel comfortable discussing or prioritize sharing during a clinical consultation. Querying the Reddit migraine subforum also provides insight into the experience of more than 54,000 unique users, which could include diverse sources ranging from patients and family members to experienced health care providers.

However, the use of Reddit data also confers significant limitations. We cannot correlate our findings with individual patient demographics, previous medical history, or other clinically relevant information. All information is self-reported and subjective, which calls into question the accuracy of unverifiable data. Although the number of posts differs between erenumab and fremanezumab due to FDA approval time, we do not expect this to significantly change our result since word distribution should be theoretically constant despite the sample size. Another limitation is the use of multiple different words for the same symptom; for example, when “lightheadedness,” “dizzy,” and “dizziness” collectively describe the same entity. Relevant symptoms described in various ways may be missed during this type of analysis. Pooling multiple similar terms could avoid this problem in future studies. Finally, our method did not analyze comments responding to specific posts; incorporating this into future studies using word frequency methodology could increase the accuracy of our conclusions. Future directions for our line of research include the analysis of other digital platforms to compare with our findings.

Postmarketing pharmacovigilance remains critical, especially when RCTs underdetect adverse events. When compared with health care professionals, direct patient reporting of adverse drug events identifies more potential signals and in greater detail [28]. Our study provides support that monitoring patient reports via social media can be incorporated into postmarketing safety surveillance for preventive migraine medications. While our proposed methodology has its limitations, the data offer a broader postmarketing perspective that complements clinical trial findings and case reports. We hope that consideration by relevant stakeholders will enhance existing tools in the pharmacovigilance armamentarium, enabling quicker and earlier detection of adverse effect signals. However, more work is needed to improve the reliability and validity of social media analysis, which faces the same challenges inherent to big data, including high volume, unstructured capture, and structural regulatory barriers, which all need to be addressed before implementation [29].