Postpartum Migraine Headache Coding in Electronic Health Records of a Large Integrated Health Care System: Validation Study

Introduction

Migraine is a common neurological disorder characterized by repeated headaches of varying intensity [1]. Often presenting in one side of the head, these headaches are typically accompanied by nausea, vomiting, and extreme sensitivity or intolerance to light and sound that lasts for 4 to 72 hours [1,2]. Globally, migraines were ranked as the sixth most prevalent cause of diseases [3] and the third cause of disabilities [4]. The prevalence of migraine increases initially with age, peaks at about 30-39 years, and then gradually declines over time [3]. Migraine headaches are more prevalent among women of reproductive ages (15-49 years), with 25% of women affected worldwide [3]. Women also experience more severe symptoms with a longer duration of attack, recurrent headaches, and higher migraine-related disability than men [5]. Moreover, migraines during pregnancy have been linked with preeclampsia or postpartum stroke [6,7]. Migraines during the postpartum period are particularly common, with reported prevalence rates ranging from 5% to 55% [8]. However, due to the lack of a validated instrument for uniform case ascertainment, there is uncertainty in the reported prevalence in the literature.

In 2009, the Health Information Technology for Economic and Clinical Health (HITECH) Act was enacted to promote the adoption of electronic health record (EHR) systems among health care providers [9]. Consequently, many health care institutions implemented EHR systems in their facilities [10]. The Kaiser Permanente Southern California (KPSC) integrates EHRs with comprehensive inpatient and outpatient clinical records, and prescription medication history can be used for pharmacoepidemiologic research, including examining potential risk factors and triggers for postpartum migraine and its effects on the lives of the affected individuals.

Currently, many health care systems use the official International Classification of Diseases-Clinical Modification (ICD-CM) coding systems to classify diagnoses and procedures in the EHR [11]. In KPSC settings, ICD-9-CM coding was transitioned to the ICD-10-CM coding system after October 1, 2015, which provides increased specificity and details for many health conditions [11]. However, the accuracy of postpartum migraine headache data in the EHRs, including the impact of the ICD-9-CM/ICD-10-CM transition, has not been fully examined, leaving uncertainty around the validity of case ascertainment methods. Furthermore, whether such accuracy improves when postpartum migraine headache diagnosis (Dx) codes are used in conjunction with automated pharmacy records (Rx) has not been elucidated. Therefore, we evaluated the reliability and accuracy of postpartum migraine headache Dx codes with and without the supplemental use of Rx and whether the ICD-10-CM coding system has improved or decreased the accuracy of postpartum migraine headache case ascertainment.

Methods

Study Setting

This study was conducted using EHR data from KPSC. The KPSC health care system comprises over 4.8 million members, 15 hospitals, and 236 medical offices throughout southern California. Prenatal and postnatal care to the member patients is provided as outpatient care at KPSC. Although most members receive their care at KPSC hospitals and <10% utilize contracting hospitals, all diagnostic, procedural, and Rx data are captured and maintained by the KPSC EHR since its full implementation in 2008. Furthermore, the characteristics of KPSC members closely reflect the Californian population [12].

Ethics Approval

This study was approved by the KPSC institutional review board (approval 13114), and informed consent was waived, as the study was low risk and strictly involved the use of internal EHR data, that is, access permission was given to authorized personnel only when needed.

Cohort and Sample Selection

Data were obtained retrospectively from women who delivered live infants at the KPSC health system during 2 distinct time periods: (1) January 1, 2012 to December 31, 2014 (ICD-9-CM period) and (2) January 1, 2017 to December 31, 2019 (ICD-10-CM period). We carefully selected the 2 time periods to investigate the medical coding accuracy before and after the implementation of ICD-10-CM in the KPSC system. For each time period, we randomly sampled 25 cases in each of the following 4 strata based on EHRs: (1) those without any Dx codes or Rx for migraine headaches (neither Dx nor Rx), (2) those with only Dx codes for migraine headaches (Dx only), (3) those with only Rx selected a priori as a treatment for migraine headaches (Rx only), and (4) those with both Dx and Rx (Dx+Rx). Thus, a total of 200 individual deliveries were selected for this validation study via the stratified sampling scheme mentioned above. The accuracy for each of the 4 strata, either case ascertainment (Dx only, Rx only, or Dx+Rx) or noncase ascertainment (neither Dx nor Rx), was expected to be around 85%. A sample size of 25 would provide less than 15% one-sided margin for a 90% CI of the accuracy for each stratum.

Defining Postpartum Migraine Headache

In this study, postpartum headache was defined based on documented clinical records within the first 12 months after delivery. In particular, primary clinical Dx codes, coded by medical coders from the clinical data management team (See Multimedia Appendix 1 for ICD-9-CM/ICD-10-CM codes) and pharmacy dispense records from all inpatient and outpatient services during the postpartum periods of the index pregnancy were used to ascertain its diagnosis (see Multimedia Appendix 1 for the medication list).

Chart Abstraction Process

Two trained research associates reviewed EHRs for documentation of a diagnosis or medication for migraine headache during the postpartum period. Women were considered to have postpartum migraine headache if either a mention of migraine headache or a prescription for the treatment of migraine headache was noted in the chart during the first year of the postpartum period. Research associates confirmed any medical diagnosis (yes/no) and prescription for treatment (yes/no) of migraine headaches for all 200 women. To ensure data quality and consistency of chart reviews between the 2 abstractors, interrater reliability assessment was performed on randomly selected charts stratified by the 4 strata (DX only, Rx only, Dx+Rx, and neither Dx nor Rx). Discrepant cases in clinical utilizations were adjudicated by the study investigators with clinical expertise (MJF and DG). The postpartum migraine headache cases abstracted through this process served as the gold standard.

Maternal Characteristics

Maternal characteristics for KPSC births included maternal age (<20, 20-29, 30-34, and ≥35 years), race/ethnicity (categorized as non-Hispanic White [hereafter referred to as White], non-Hispanic Black [hereafter referred to as Black], Hispanic, Asian/Pacific Islander, and others/unknown), educational attainment (less than high school, high school graduate, some college, bachelor’s/associate degree, and master’s degree or higher), household income in USD (<$30,000, $30,000-$49,999, $50,000-$69,999, $70,000-$89,999, and ≥$90,000), timing of prenatal care (early or first trimester and none or late initiation), and self-reported smoking (yes/no). Gestational age at birth, reported in completed weeks, was derived from clinical estimates.

We obtained the characteristics of all births of State of California residents during the same time periods by using publicly available data that has been posted on the Centers for Disease Control and Prevention Wonder website [13]. Both the KPSC EHR and the Centers for Disease Control and Prevention Wonder website provided information on maternal characteristics, including maternal age, race/ethnicity, educational attainment, timing of prenatal care, smoking during pregnancy, and gestational age at delivery (in completed weeks of gestation). Data on median household income was estimated based on census tracts for KPSC deliveries.

Statistical Analysis

We described the characteristics of our study population, all women who delivered in KPSC hospitals, and the State of California residents with live births during 2012-2014 and 2017-2019 by using frequency distributions. We also calculated the kappa statistic (κ), which estimates agreement between the 2 abstractors. As mentioned above, abstracted chart reviews for migraine headache cases were set as the gold standard. We compared findings from the manual chart review with corresponding diagnosis and prescription medication utilization records for ICD-9-CM and ICD-10-CM coding periods through sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). These performance measurements were reported as weighted percentages with corresponding 95% CIs using normalized sampling weights (W_i, i=1,2,3,4), derived by dividing the number of deliveries in each stratum by the multiplication of the total study population and the number of samples from the corresponding stratum. To evaluate the overall performance, we also reported the summary statistics of F-score and Youden J statistic (J), which are composite measurements of sensitivity and PPV (F-score) or sensitivity and specificity (J). All analyses were conducted using SAS statistical software version 9.4 (SAS Institute Inc).

Results

An overview of the patient characteristics for our sample, study population, and California state birth population is shown in Table 1. Overall, 157,501 deliveries from all KPSC hospitals were obtained during the 2 periods, with 72,471 for 2012-2014 and 85,030 for 2017-2019. Compared with California State birth data, KPSC deliveries had slightly higher percentages for maternal age over 30 years (1,465,998/2,874,396, 51% vs 96,157/157,501, 61.05%, respectively), education with college and above (1,605,882/2,874,396, 55.87% vs 114,297/157,501, 72.57%, respectively), and early prenatal care initiation (first trimester; 2,386,232/2,874,396, 83.02% vs 147,017/157,501, 93.34%, respectively). There were some discrepancies for race/ethnicity, likely due to the unknown information for 30.07% (864,311/2,847,396) of the state population versus only 3.67% (5784/157,501) for the KPSC population. Despite the stratified sampling, our sample of 200 deliveries was broadly representative of the KPSC study population with respect to maternal age, early prenatal care initiation, prenatal smoking status, and gestational age. However, Hispanics and lower household income populations (<US $69,999) were slightly oversampled.

The overall κ between our abstractors was 100%. Table 2 gives the distribution and their sample sizes for the 4 strata (Dx+Rx, Dx only, Rx only, and neither Dx nor Rx) among our overall study population and by the 2 time periods. The majority (150,801/157,501, 95.75%) of the women did not have any record of diagnosis or pharmacy usage, while only 1131 (0.72%) women had Rx indicating obtaining prescription medication but without a diagnosis of postpartum migraine headache. Regardless of the uneven numbers among the 4 strata, we sampled 50 cases from each stratum. The corresponding chart review results and the normalized weights are provided in Table 2. For those with both Dx codes and Rx, our chart abstractors confirmed all 50 had a postpartum migraine headache, and those with neither were all confirmed to be noncases. The sample with postpartum migraine headache Dx codes only had 88% (44/50) of true positives, while having prescription Rx only resulted in 96% (48/50) of true positives.

Based on these comparisons against the chart review results, the performance measurements of our EHR are shown in Table 3. The overall weighted sensitivity, specificity, PPV, and NPV for postpartum migraine headaches were 100%, 99.7%, 93.9%, and 100%, respectively, if the postpartum migraine cases were determined by having either Dx or Rx. The corresponding F-score and Youden J statistic were 96.9% and 99.7%, respectively. Compared with using Dx codes alone (either Dx+Rx or Dx only) for postpartum migraine case identification, using Rx alone (either Dx+Rx or Rx only) had poorer performance, especially on the sensitivity (82.7% for Dx codes alone vs 57.5% for Rx alone). Results were similar across the ICD-9-CM and ICD-10-CM coding time periods.

Discussion

This validation study was performed to determine the accuracy of identifying postpartum migraine headache cases by using data abstracted from the EHR of a large health care system with a sociodemographically diverse patient population. To our knowledge, the accuracy of the data on postpartum migraine headache using both Dx and Rx has not been validated in EHR data or the extent that the transition of ICD-9-CM to the ICD-10-CM coding system has impacted postpartum migraine headache case ascertainment. Our study showed that having either Dx or Rx had high sensitivity (100%) and specificity (99.7%) for case ascertainment. The overall performance measured by the F-score and Youden J statistic were 96.9% and 99.7%, respectively. Such findings were similar across the ICD-9-CM and ICD-10-CM coding time periods.

Over the past few decades, EHRs have become important data sources for pharmacoepidemiologic studies and have become standard among health care providers as part of the American Recovery and Reinvestment Act of 2009 (specifically, the HITECH Act) [9]. Although the EHR is a highly sophisticated information management and care delivery system that ensures quality care by providing access to comprehensive patient information and the latest best practice research, its completeness and reliability for pharmacoepidemiologic studies have been questioned due to various reasons, including clinical knowledge, attention to details, communication between providers and coders, coding procedures, and others [14-16]. At KPSC, the process of coding and coding rules of the medical diagnosis recorded in patients’ EHRs is carried out by highly skilled medical coders from KPSC’s clinical data management team. Furthermore, the individual medical coder’s accuracy has been evaluated critically for consistency. Nevertheless, we may still need to develop reliable disease-specific algorithms by using diagnosis coding in combination with other EHRs, including but not limited to pharmacy and utilization records [17]. Therefore, we performed this validation study to evaluate (1) the accuracy of postpartum migraine headache case identification in EHRs and (2) the impact of implementing the ICD-10-CM coding system in extracting data on patients with postpartum migraine headache in the health system. The findings of this study suggested that the validity of EHR data for the identification of postpartum migraine headache cases differed depending on the EHR data sources (Dx and Rx). The accuracy of case ascertainment based on Dx was higher than that of case ascertainment based on Rx in this study. We speculate that the low performance of pharmacy data on case ascertainment may be driven by lactating patients declining migraine medication to avoid infant exposure through breastmilk. Others may have opted to use over-the-counter pain medications rather than prescription medications. It is also possible that a given prescription may be an appropriate drug for conditions other than migraine. However, postpartum migraine headache cases can be identified with high accuracy if Dx codes and prescription medications for the conditions are used together. Furthermore, the transition from the ICD-9-CM coding system to the ICD-10-CM coding system had minimal impact on the overall accuracy of ascertaining postpartum migraine headache cases.

The main strength of this study is the development of a valid and reliable method of postpartum migraine headache case ascertainment by using EHR data extracted from a large insured and socioeconomically diverse southern California population, which is likely generalizable to other health care settings with similar EHR database systems. In addition, in this study, the review process includes the entire medical health record, not limited to clinical Dx codes and prescription Rx, for determining the accuracy of postpartum migraine headache. A potential limitation of this study was the use of medical record abstractors who were not blind to the source of the data. Although it is possible that this could have biased the study in unforeseen ways, a previous study that evaluated the agreement between masked and unmasked medical records abstractors reported no impact of this [18].

In conclusion, our findings suggest that postpartum migraine headache is not reliably coded in the EHRs. The use of Rx along with clinical Dx codes improves the identification of true postpartum migraine cases more than the use of clinical Dx codes alone, and the transition to the ICD-10-CM diagnosis coding system had no impact.