EHR data were extracted and analyzed for the study period (June 1, 2016, to June 1, 2022). The study population included all patients aged 6 to 18 years [15,19] who visited the ED during the study period, identified through the presence of an ED note. Patients were screened for suicide using the Ask Suicide-Screening Questions toolkit [20]. Patient EHR data were deidentified and loaded into Cumulus (Computational Health Informatics Program) [21], a platform for population health studies [22,23].

An ICD-10–based suicidality case definition [5] was used to select patient cohorts who matched suicidality criteria. Comorbidities in the suicidality population were reviewed as potential study variables for the ICD-10–based suicidality case definition. Statistically significant associated comorbidities were then used as input variables for a propensity score matching (PSM) model to select patients who did not match an ICD-10–based suicidality case definition [24]. The propensity-matched cohorts were compiled for chart review. The accuracy of the comorbidity-enriched PSM model was then compared to the interpretation of 2 subject matter experts (RX and AJM).

Figure 1 shows the sequence of case cohort selection, comorbidity analysis, comparator cohort matching, and chart review validation. First, cohorts were selected using an ICD-10–based suicidality case definition. Second, suicidality comorbidities were selected using conditional probabilities, tested for significance, and corrected for multiple hypothesis testing. Third, PSM was used to select patients similar to known suicidality cases. Fourth, manual chart review was used to assess the PSM model identification of suicidality cases missed by conventional ICD-10–based case selection.

This retrospective observational study was approved by the Boston Children’s Hospital institutional review board (IRB-P00043211) with a waiver of ethics approval and informed consent. Patient data were deidentified before analysis.

Diagnosis of suicidality included 3 subtypes with increasing severity: ideation, self-harm, and attempt [6,24]. ICD-10 codesets were compiled for each suicidality subtype [5]. Suicidal ideation is documented only for the present encounter (R45.851) as there are no ICD-10 codes for history of suicidal ideation. Self-harm has 4 codes for personal history (R45.88, Z91.5, Z91.51, and Z91.52) and 1396 codes in total. Suicide attempt includes 4 codes: suicide attempt (T14.91), initial encounter (T14.91XA), subsequent encounter (T14.91XD), and sequela of suicide attempt (T14.91XS).

Study variables included the ICD-10–based suicidality case definition, ICD-10–based diagnosis codes for comorbid DSM-5 disorders, patient demographics, encounter service date, health care use, and physician note types. Demographic variables included patient-reported race, sex assigned at birth, and age at ED visit, with age groups categorized as children (6‐11 years) and adolescents (12‐18 years). The encounter service date was used to determine the encounter sequence. Health care use provided a measure of the total number of patient visits and the number of clinical note types per patient. Each encounter was linked to clinical notes including the ED note and, optionally, psychiatric notes and discharge summary. The statistically significant codes resulting from the comorbidity analysis were added to the list of study variables (Multimedia Appendix 1).

Conditional probability was used to quantify the relationships between additional ICD-10 diagnoses and the ICD-10–based suicidality case definition. The conditional probability of suicidality A given the concurrent presence of a comorbid condition B in the same patient (equation 1) is denoted as follows:

A denotes the suicidality ICD-10 codeset. B denotes a particular ICD-10 diagnosis not included in the suicidality codeset. P(A ∩ B) signifies the probability of suicidality A and diagnosis B occurring in the same patient, and P(B) denotes the independent probability of another ICD-10 diagnosis code B. Expressed within the range of 0 to 1, a value of 0 signifies that the diagnosis pairs never co-occur, whereas a value of 1 indicates that the diagnosis pairs always co-occur. Conditional probabilities were calculated using equation 1 for all possible co-occurring ICD-10 diagnosis codes stratified by age group, sex assigned at birth, and suicidality subtype (ideation, attempt, and self-harm). To increase the reproducibility and practical utility of the comorbidity model, only the 20 top-scoring co-occurring ICD-10 codes were selected for each stratification. We then tested the statistical significance of these associations for each stratifier using a chi-square test for independence, with a Bonferroni correction applied to control for false discovery of multiple comparisons (SciPy; version 1.15.2). ICD-10 codes occurring in fewer than 20 patient cases were omitted from the analysis to reduce the effect of rare events [25]. Conditional probabilities and ICD-10 diagnosis pairs are further referred to as comorbidities even when not all ICD-10 codes necessarily indicate a clinically recognized comorbid condition.

PSM was used to match the case cohort to a comparator cohort covariates across all study variables, that is, patient demographics (age group, biological sex, and race), health care use codes, clinical note types, DSM-5 categories, and significant ICD-10 codes selected from the comorbidity analysis (Multimedia Appendix 1). The case cohort included every patient encounter with a suicidality diagnosis matching the case definition. The comparator cohort included every other patient encounter during the study period excluding patients matching the case definition. Case and comparator cohorts were compiled and ordered by encounter date, aggregating study variable observations up to the date of the encounter. To attain larger sample sizes and reduce variation in ICD-10 billing practices, ICD-10 codes were grouped using the DSM-5, Text Revision [9].

The study variables were compiled into a feature matrix [26] and analyzed using the PsmPy Python package [27]. A logistic regression model estimated propensity scores for suicidality using the feature matrix as input. Similarity between suicidality cases and patients without a suicidality diagnosis was measured using the PSM logit score (ie, the logarithmic transformation of the predicted probability and k-nearest neighbor).

The source code [28] for reproducing this work is freely available under an open-source Apache 2.0 License.

During the study period, there were 59,866 patients aged 6 to 18 years with 90,980 ED encounters in total, of which 2638 (2.9%) encounters matched an ICD-10–based case definition of suicidality. Table 1 reports demographic characteristics for all ED encounters stratified for each suicidality subtype. Suicidality was significantly more common among female than male individuals across all suicidality subtypes (P<.001 via chi-square test).

Table 2 shows the ICD-10–based prevalence of mental health conditions observed in all ED encounters according to DSM-5 categories. Anxiety disorders were the most prevalent (9343/59,866, 15.6%), followed by attention-deficit/hyperactivity disorder (6250/59,866, 10.4%) and depressive disorders (5426/59,866, 9.1%). Notably, gender dysphoria (333/59,866, 0.6%), dissociative disorders (232/59,866, 0.4%), and personality disorders (190/59,866, 0.3%) were rare DSM-5 categories.

Comorbidity analysis revealed that the conditional probability of suicidality was highest for the following DSM-5 categories: personality disorders (84/190, 44.2%), gender dysphoria (143/333, 42.9%), bipolar disorder (162/448, 36.1%), depression (1842/5426, 33.9%), and schizophrenia spectrum disorders (133/411, 32.4%). Conversely, the probability of DSM-5 category–given suicidality was highest for depression (1842/2230, 82.6%); anxiety (1597/2230, 71.6%); trauma- and stressor-related disorders (802/2230, 36.0%); attention-deficit disorder and attention-deficit/hyperactivity disorder (721/2230, 32.3%); and disruptive, impulse control, and conduct disorders (609/2230, 27.3%). Nearly half (1127/2230, 50.5%) of all patients with an ICD-10 code for gender dysphoria (F64.8) also met the suicidality case definition. Figure 2 illustrates the conditional probabilities of suicidality and DSM-5–categorized mental health conditions. Detailed statistics for each DSM-5 category can be found in Multimedia Appendix 2.

Of the 55 statistically significant ICD-10 suicidality risk factors, nearly half (n=26, 47.3%) were not part of the DSM-5 (Multimedia Appendix 1). Nearly a quarter (n=12, 21.8%) were for harm without documented self-harm intent, including “unintentional” drug poisoning and “accidental” lacerations of arms and legs. These ICD-10 codes for unintentional poisoning or accidental injury were strongly associated with self-harm diagnosis. Female patients were significantly more likely (odds ratio 2.23, 95% CI 1.61‐3.10; P<.001) than male patients to have ICD-10 codes related to accidental (unintentional) or undetermined self-harm. Multimedia Appendix 3 provides a more detailed report on findings between subpopulations.

Mental health conditions such as F48.9 (nonpsychotic mental disorder, unspecified), F43.12 (posttraumatic stress disorder, chronic), and R44.0 (auditory hallucinations) were not found in DSM-5 classifications [9], yet they were significant risk factors for suicidality within the study population. A full list of the top-scoring risk factors for suicidality is provided in Multimedia Appendix 4.

PSM was used to match cases with comparators sharing a similar comorbidity burden that may underreport cases using an ICD-10–based case selection. Figure 3 illustrates the distribution of propensity scores for suicidality cases and comparators before and after matching. Propensity scores closer to 0 indicate lower likelihood of suicidality, whereas scores closer to 1 represent a higher likelihood. Before matching, 70.4% (1571/2230) of suicidality cases (Figure 3) had a propensity score of ≥0.9. In contrast, only 1.9% (1051/56,725) of comparators had scores of ≥0.9 (Figure 3). After matching, the distribution of matched comparators (Fig. 4d) more closely resembled that of the cases. The number of comparators with propensity scores of <0.5 dropped from 94.7% (53,729/56,725) before matching to 12.2% (270/2230) after matching, demonstrating that PSM was effective in aligning the comparator group’s comorbidity profile with that of the suicidality cases. Multimedia Appendix 5 provides more details on the PSM analysis.

Propensity-matched cohorts were selected for chart review (n=205 encounters) by subject matter experts [5]. Compared to the prevalence of suicidality in the overall study population (2638/90,980, 2.9%), the prevalence in a propensity-matched cohort was 22.3% (23/102) per human expert review. The difference in proportions of suicidality in the 2 populations was significant (P<.001; 2-proportion z test).

In this study, a patient-matching method using comorbidities as covariates identified a cohort of patients with high prevalence of suicidality despite lacking ICD-10 codes for suicidality. This finding has several important implications. First, the PSM methodology could enable more accurate cohort identification for suicidality research; public health surveillance; and population-level analyses of suicidality risk, prevalence, and outcomes. Currently, the ability to identify suicidality cases relies heavily on ICD-10 codes for suicidality, which are far from comprehensive and miss many cases [5-8]. Our established methodology incorporated both DSM-5–related codes and ICD-10 codes not related to the DSM-5, expanding the range of comorbidity patterns associated with suicidality. Non–DSM-5 codes, such as accidental poisoning or injury-related encounters, were statistically significant, indicating that relevant risk factors extend beyond formal psychiatric diagnoses. By demonstrating the utility of this broader, comorbidity-informed PSM approach to identify potential underdiagnosis and/or inadequate documentation of suicidality, this study advances methods for improving case ascertainment, including unrecognized cases of suicidality (ie, those that are not labeled with an ICD-10 code for suicidality yet have demonstrated evidence of suicidal behavior).

These findings have important clinical implications as well. Mental illness and suicidality are a growing problem among youth in the United States. The ability to identify these cases accurately is essential both for clinicians to provide appropriate mental health care and reduce suicidality among those at risk, who may currently go unrecognized, and for researchers to study these problems and generate insights that can inform public health resource allocation and intervention development. This study identified comorbidity patterns that are associated with increased probability of suicidality. In patients with these phenotypes, additional screening may be warranted to assess for suicide risk. For instance, patients with multiple ED encounters for supposedly unintentional self-harm may be at increased risk of suicidality and require more in-depth psychiatric evaluation than the standard suicide screening measures used for all ED patients.

Suicide attempts were identified among preadolescent patients (<12 years), with a prevalence of 2.8% (5/177). The presence of suicide attempts in this younger age group underscores that suicidal behavior can emerge earlier in development than traditionally recognized. This observation is consistent with recent evidence reporting measurable rates of suicidal ideation and attempts in preadolescents [7,10] and emphasizes the need for early detection strategies and age-specific prevention efforts within pediatric clinical care. Notably, subclinical personality pathology in the developmental age (ie, personality disturbances) may amplify the effects of co-occurring psychiatric conditions, contributing additively to suicidality risk [29] and highlighting vulnerabilities that extend beyond what can be captured by categorical diagnostic frameworks such as the ICD-10 and DSM-5. Similar to suicidality, personality pathology may be underrecognized in pediatric populations due to these diagnostic limitations.

Moreover, while the objective of this study was to develop a method to identify unrecognized cases of existing suicidality rather than predict future risk of suicidality [30], it is feasible that the high-risk comorbidity patterns identified in this study are also associated with increased risk of future suicidality. Additional research is needed to understand how different comorbidity patterns predict suicidality risk and how such insights could be operationalized to improve care and outcomes. Future research could also examine how developmental processes interact with conditions such as personality pathology and gender dysphoria in shaping suicidality risk. These interactions may be more accurately captured through dimensional assessment frameworks that evaluate personality functioning and emotional regulation along a continuum rather than through categorical diagnoses [30].

Public health surveillance of pediatric suicidality using ICD-10 codes is likely underreported. In the chart-reviewed sample, only approximately half (53.4%) of suicidality cases were identified using ICD-10 codes. Recent pediatric suicidality studies have reinforced the need for computable phenotypes beyond a traditional ICD-10–based case definition of strict diagnostic criteria, namely, machine learning [7,31] and natural language processing [32,33] methods. While it was beyond the scope of this study to estimate the true and apparent prevalence [34] of suicidality, it may be possible to use large language models to provide a more accurate prevalence estimation [35].

There were limitations in this retrospective observational study using EHR data. First, this study was conducted at a single large pediatric medical center in a northeastern urban area of the United States. Clinically relevant comorbidity profiles may vary by geography and urban vs rural settings. Second, this study focused on suicidality in ED encounters. ICD-10 coding practices may vary across ED, inpatient, and ambulatory settings. Third, while PSM identified many instances of suicidality that were missed using ICD-10 codes alone, it still might not identify all suicidality cases. Fourth, the effectiveness of PSM is highly dependent on the selected covariates [36]. Unmeasured or uncoded diagnoses could not be accounted for, potentially limiting the accuracy of matched comparisons. To maintain clinical interpretability and feasibility, we restricted the model to top-ranking comorbidities that a clinician could reasonably monitor in practice. However, other significant comorbidities not included in the model may also influence suicidality risk and affect matching accuracy. Further studies to test and refine the methodology may help increase the case detection yield.

This study provides a method for selecting case-matching cohorts with pediatric suicidality. Variables for the matched selection included significantly associated suicidality comorbidities and demographic variables in the study population. These findings suggest that pediatric suicidality may be highly underreported using an ICD-10–based case definition of suicidality.