Mental disorders are one of the leading public health problems worldwide [1-3], yet they are often underdetected or misdiagnosed, particularly in underrepresented groups (sexual, gender, racial, and ethnic minorities) and young adults [4-8]. The stigma of mental illness [9], concerns about confidentiality, shame, or a variety of other reasons lead some individuals to avoid explicit help-seeking behaviors that could alleviate the symptoms of mental disorders [10]. Some turn to social media and web-based resources to share and better understand their mental health symptoms and experiences [11].

In recent years, social media has become a rich source of data for studies of mental health communication, disclosure, prevalence rates, and trajectories [12]. In the medical field, more generally, data science approaches have allowed researchers to mine large health care data sets from social media [13]. In a systematic review of how social media data have been used to classify and predict the mental health state of users [14], text analysis was found to be the most common analytic method across 48 studies, in particular, the use of natural language processing and unsupervised machine learning algorithms to analyze patterns of term use. For example, ten Thij et al [15] searched on Twitter (Twitter Inc) for individuals who had reported receiving a diagnosis of depression on the platform and compared the timing of their posts with those of a random sample of Twitter users, finding differences in the pattern of activity by time of day. Specifically, the individuals who reported depression diagnoses showed more web-based activity during later hours of the day. Similarly, Bathina et al [16] showed the frequency of occurrence of cognitive distortion schema, sequences of 1 to 5 words associated with the expression of cognitive distortions, in the language of individuals who reported a clinical diagnosis of depression on Twitter versus in the language of a random sample of individuals.

Analyses of social media data, such as the works of ten Thij et al [15], Bathina et al [16], and Coppersmith et al [17], have historically relied on individuals’ self-disclosure of mental health status or search terms related to mental health disorders, for example, someone tweeting “I was just diagnosed with depression by my doctor.” A key limitation of these approaches is self-selection bias, which results, in part, from relying on people’s willingness to reveal stigmatized identities. In addition, people who are willing to self-disclose their mental health status on a public platform may be less inhibited or more likely to seek social support but may not be representative of all those who struggle with their mental health. For example, in 2014, De Choudhury and De [18] studied the self-disclosure of mental health problems on Reddit. On Reddit, individuals discuss a variety of mental health concerns, ranging from “daily grind” to highly specific questions about their diagnoses and treatment. An analysis of subreddits on “mental health,” that is, Reddit channels on specific topics, showed that individuals who are a part of subreddits related to mental health used more negative emotion words and more frequently disclosed challenges of daily activities than participants (“redditors”) in more general Reddit channels. In addition, anonymous redditors tended to be more disinhibited and share more details about their personal relationships and health issues in casual terms that did not reflect their current mental health status. In summary, anonymous Reddit accounts (“throwaway Reddit accounts”) showed greater levels of self-disclosure related to mental health and decreased feelings of vulnerability [18].

Although text analysis is the most commonly used method to study mental disorders at scale, the reliability and validity of web-based data have been called into question, particularly with respect to self-reported diagnoses. Although we have reason to believe that individuals tend to be relatively accurate reporters of their mental health experiences [19], relying on text analysis alone may be perceived as insufficient to establish ground truth. Although clinician reports may provide an alternative to self-reports, they are more time intensive to collect at scale, require access to care, and are costly [20,21]. These concerns warrant a multimodal approach that combines language modeling, sentiment, clinical diagnostics, and user engagement to reliably study common psychiatric disorders from social media data [22,23]. Crucial to many of these efforts is the ability to triangulate self-reported diagnostic information with clinical indicators. This is the idea that motivated our study, which asked the following question: Do self-reported diagnoses correspond to validated disorder-specific severity questionnaire scores?

A better understanding of the degree of correspondence between self-reported diagnoses and clinical indicators will afford researchers and clinicians higher levels of trust in self-reported diagnoses on social media. This is particularly the case for a common approach that relies on web-based statements of self-reported diagnoses to establish clinically relevant samples [15-18].

The primary aim of this study was to examine whether people are “accurate” when they report that they are or should be diagnosed with a mental disorder. In other words, does an individual’s belief that they should be diagnosed with a disorder correspond to their scores on validated self-report measurements of their symptoms. We addressed this research question using two large-scale samples of thousands of individuals: (1) a nationally representative sample of adults and (2) a large sample of university undergraduates. The participants were asked to fill out self-report questionnaires on a variety of mental disorder symptoms and asked whether they had ever been diagnosed with any conditions or felt that they should be diagnosed with those. We first hypothesized that there would be a positive relationship between self-reported disorder-specific severity scores and having received a diagnosis. Second, we expected that there would be significant differences between the participants who believed that they should receive a diagnosis and those who reported that they had never been diagnosed. Finally, we hypothesized that the group of participants who believed that they should be diagnosed would look more similar to the participants who had been diagnosed than to those who had never been diagnosed, lending initial credibility and support for relying on self-diagnoses in future research.

This study was approved by the institutional review board of Indiana University (#2002549202 and #2005948214).

The participants were recruited for a broader study called the Survey Online Cohorts for Internalizing Symptoms and Language. The aim of the Survey Online Cohorts for Internalizing Symptoms and Language studies was to triangulate social media data with self-reported mental health status. In this manuscript, we focused on the latter source of data. The participants were sampled from two distinct sources: (1) a national Qualtrics panel of Twitter users (study 1) and (2) a psychology student pool at a large Midwestern university in the United States, the participants from which received credit for participation (study 2). In both studies, the participants completed a battery of questionnaires entirely on the web. Our inclusion criteria consisted of passing reCAPTCHA and affirming “Yes” to the question, “I will provide my best answers.” After excluding the individuals who could not pass the reCAPTCHA or did not commit to providing truthful answers, a total of 3460 participants responded to our surveys. Although the procedural elements and questionnaires completed by the participants were nearly identical, we have first presented the nationally representative sample of Qualtrics participants (study 1) and then presented the college student sample (study 2). Data were collected between July 2020 and April 2022. Additional data collection is ongoing for facets of a larger project unrelated to this study.

The demographic information of the participants of studies 1 and 2 is presented in Table 1. For study 1 (n=1123), our sample was well represented, with the participants’ racial demographics mapped onto census data and a large distribution of age. As for study 2 (n=2337), it can be assumed that the sample was not nationally representative, as most participants were Indiana residents who were college aged.

The data were analyzed using R Studio (version 1.0.153, RStudio, Inc). For each diagnosis (eg, depression, anxiety, and substance use) we conducted an ANOVA test to determine whether the disorder-specific severity questionnaire scores were different among the 4 diagnostic groups (ie, “Yes”; “No, but I should be”; “I don’t know”; and “No”). We used Tukey post hoc tests to determine whether the differences in means between the groups were statistically significant.

Table 2 presents the number of participants from each study who selected (1) “Yes”; (2) “No, but I should be”; (3) “I don’t know”; and (4) “No” to the question, “Have you ever been clinically diagnosed with [blank] disorder,” along with the percentage of the sample they represented. As shown in Table 2, study 1 had higher prevalence rates of each disorder assessed, indicating more clinical diagnoses in the national sample than in the college student sample. Missing data are not included in Table 2, but approximately 1% to 2% (from 11 to 19 participants in study 1 and 30 to 31 participants in study 2) did not answer the questions regarding diagnoses.

This study investigated the scores on self-reported disorder-specific severity questionnaires of individuals who indicated that they had been previously diagnosed (“Yes”), believed that they should be diagnosed (“No, but I should be”), were unsure (“I don’t know”), and had never been diagnosed (“No”) across a variety of internalizing and externalizing disorders. Given the misdiagnosis, underdetection, and stigma associated with mental disorders, this study was a novel way to explore the “accuracy” of self-diagnosis in a national survey panel and a sample of college students. Overall, the results indicate that people who say they have been diagnosed with a disorder have the highest severity scores in general, but this is closely followed by individuals who have not been diagnosed but believe that they should be. This is especially true for clinical conditions that are relatively nonspecific and whose names are relatively descriptive such as depression, generalized anxiety, and insomnia. The results suggest that self-reported mania questionnaires do not align with reported diagnoses, which is consistent with prior studies [33-35], although there is some evidence that self-reported measures can correlate with a lifetime diagnosis of bipolar disorder [36]. Additional differences in patterns observed between our samples contribute to our interpretation of these findings; with some exceptions, college students tended to be more accurate than the national sample in self-diagnosis.

Although study 2 consisted of college students in the Midwestern United States, which was a relatively homogeneous population in terms of age and ethnicity, our national sample (study 1) was more variable in age, ethnicity, and other demographic characteristics, including age and gender. Our national sample from study 1 also reported more severe psychopathology. The differences between our samples, with college students being slightly more accurate in self-diagnosis, yield additional support for the validity of university samples, which are often criticized for lacking generalizability [37]. Initially, we had expected that college students may not know as much about their diagnoses, but it seems that college students are reasonably accurate in self-diagnosis, reflecting trends of reduced stigma and increased awareness of the mental health concerns that are present in early adulthood [38,39]. In addition, it is possible that the college students were more accurate in self-diagnosis because they were psychology students and, therefore, were aware of various mental health complaints. A secondary interpretation is that younger adults are generally better at web-based help-seeking behaviors and thus more attuned to their personal mental states [40]. Indeed, younger adults may be more prone to and more savvy with engaging in web-based information seeking, including the use of social media support networks to understand their mental health status [41]. This makes social media a useful tool for studying mental health as well as disseminating support resources, treatment options, and research.

Overall, our hypotheses were supported. Our findings showed that self-diagnosis is accurate for internalizing disorders, meaning that most individuals with disorders such as major depression and GAD can accurately detect that their symptoms are significant, even if they have not received a formal diagnosis. Our findings also reveal that self-reported diagnoses are more accurate for more commonly occurring conditions; for example, depression and GAD showed high accuracy in self-diagnosis, but they are also among the most common mental disorders [42,43], unlike other anxiety disorders like panic that occur at lower frequencies [43,44]. It is possible that the accuracy of self-diagnosis corresponds best to disorders that occur at high base rates, which could partially explain our poor self-diagnosis accuracy for mania, panic, and somatic symptom disorders. Indeed, prior research has demonstrated that self-reporting is more accurate for people who are more familiar with a disorder and its symptoms [45]. Social media may play an important role in disseminating information about common mental health disorders to communities of interest such as adolescents and young adults.

Interestingly, consistent with prior work, the participants in both samples showed poor abilities in detecting mania symptoms via self-report compared with diagnoses from professionals. We acknowledge this is not the first study to find that mania is more difficult to detect than depressive symptoms in a self-report form [33]. It is likely that bipolar disorder, specifically manic symptoms, cannot be screened via self-report, and a multistage screener with clinician report may be necessary [33-35]. It is also possible that the time frame of the ASRM assessment, that is, the past week, does not align well with the diagnosis of bipolar disorder, for which a manic episode could have occurred years ago [46]. Indeed, for most patients with a bipolar-spectrum illness, mania symptoms are rarely present at the time of the assessment.

Alcohol and substance use disorders followed bipolar disorder in terms of lowest self-diagnosis accuracy. We offer several explanations for this finding. First, in college students, alcohol consumption, particularly binge drinking, is much more prevalent than in the general public [47,48], and the behavior is normalized and encouraged in many social groups [49,50]. This may lead to a low likelihood of reporting interference from drinking, despite a high level of alcohol use. However, this is likely not the case in our studies, as in our college student sample, the participants who believed that they should be diagnosed actually had higher AUDIT scores than those who had already been diagnosed, indicating good insight. However, given that the rates were higher in those who believed that they should be diagnosed than in those who had received diagnoses, it is possible that college students who are struggling with alcohol and drug problems receive diagnoses for other comorbid conditions (depression) and do not receive substance use disorder diagnoses until they are out of college when heavy drinking behaviors are not normalized. Indeed, in our college student sample, <1% had been diagnosed with an alcohol use disorder, whereas in the national sample, approximately 10% had previously received a diagnosis, which is in line with epidemiological data [51]. In our national sample, which had lower socioeconomic status and may have had lower educational attainment overall compared with our university sample, there was a lower correspondence between self-diagnosis and self-report disorder-specific severity questionnaire scores, indicating a need for further research into drinking and substance use behaviors, perhaps via more daily monitoring of use.

Despite the strengths of our study, including its novelty and the use of 2 large samples, it is not without its limitations. First, we did not have clinicians interview the participants to assess their disorders at the time of the study. The reported disorders were based on prior clinical diagnoses and relied on participants’ honesty and awareness of their mental health conditions. Moreover, we did not include an assessment of all disorders, including those that may resemble mood disorders early in their course, including psychotic disorders, personality disorders, and medical or substance-induced disorders, as we focused most thoroughly on the internalizing spectrum. Future research should consider assessing the full spectrum of psychopathology. In addition, we note that the timeline of disorder onset is not clear from our questionnaires. Some participants may have received their diagnoses years before the current assessment period, and our questionnaires asked about their symptoms within the last week or 2 weeks. Second, differential diagnosis was nearly impossible in our sample, given the wide variety of disorders and lack of clinical reports. This means that the participants may have appeared to have a higher degree of psychopathology than was truly representative, given their diagnostic comorbidities, DSM hierarchy rules, and rule-outs that clinicians would be aware of but patients may not be aware of. Third, it is possible that in our study, the participants appeared to be good at yielding accuracy in the self-diagnoses of some disorders such as social anxiety disorder, but this is a reflection of the measure being highly sensitive but not specific. Further testing of the DSM severity measures in a wide variety of samples is warranted before their extended use.

Our results support ongoing and future research that uses patient self-report of symptoms, diagnoses, and disorders on a more solid footing. In particular, for some conditions of common mental disorders such as depression, anxiety, and insomnia, relying on textual reports may not bias samples, at least in terms of symptom severity: when individuals say that they struggle with a problem, they in fact seem to do so. The next steps of this research involve examining these results in conjunction with relevant features of individuals’ social media accounts, including sentiment analyses, text analysis of themes associated with internalizing disorders, and other computational informatics analyses of social media data. Indeed, we have previously shown that individuals who self-disclose depression diagnoses use more cognitive distortions [16], have different circadian rhythms [15], and show more negative affect based on sentiment analysis according to unpublished data from Rutter et al (Rutter, LA, unpublished data, August 2022). However, the validity of results from Twitter have been called into question because self-disclosure is not well understood and is often stigmatized as a nonefficacious source of data [52]. It is also possible that individuals who self-disclose their diagnoses on the web are different from those who report their diagnoses and symptoms via survey. On the basis of the results of this study, we can now more definitively say that self-reported survey diagnoses correspond well with symptom severity on a continuum and can, on balance, be trusted as clinical indicators, especially in internalizing disorders such as depression and GAD. However, more research on the validity and modalities of web-based self-reports is needed, and research should be expanded to capture other related mental disorders. Gaps in research notwithstanding, if more studies ask about self-reported diagnoses and compare self-reported symptoms, the field could benefit by receiving more accurate detection rates. In general, if a person believes that they should be diagnosed with an internalizing disorder, they may be experiencing a degree of psychopathology that is similar to that experienced by someone who has been diagnosed, warranting clinical attention. Psychologists and providers can put more confidence into the patient’s perspective of their disorders and when time is limited, reduce questionnaire questions and query more specifically if the person believes that they have a disorder. In addition, researchers may be able to place more trust in social media data when individuals report diagnoses of anxiety and depression; however, replication and further study are needed. This perspective may yield additional insight into the burden of mental illness and the associated distress and impairment.