The establishment of the MIMIC-3 database was approved by the institutional review boards (IRBs) of Beth Israel Deaconess Medical Center and the Massachusetts Institute of Technology [4]. Under the Common Rule (45 CFR 46), records-based research using identifiable private information that is publicly available is exempt from IRB review. Since MIMIC-3 is publicly available and deidentified, the secondary analysis in this study is exempt from IRB review. As a result, this study did not undergo an IRB review.

One of the authors (GI) received the necessary training in the use of the MIMIC-3 data set, obtained permission to use the data, and conducted this study in compliance with the PhysioNet Credentialed Health Data Use Agreement 1.5.0.

In this study, we used the MIMIC-3 data set, a publicly available large-scale EMR data set. MIMIC-3 contains comprehensive clinical data from over 40,000 patients admitted to ICUs at the Beth Israel Deaconess Medical Center in Boston, Massachusetts, between 2001 and 2012. The data set includes various types of clinical information such as demographics, vital signs, laboratory results, medications, diagnoses, and free-text data in the form of nursing notes, radiology reports, and discharge summaries [3-5].

This study included patients with K pneumoniae or E coli detected in bacterial culture tests conducted during hospitalization and detailed in the MIMIC-3. Patients without a summary text at discharge were excluded from this study.

The outcome variable is a binary variable of whether K pneumoniae or E coli in the specimen showed a minimum inhibitory concentration ≥8 µg/mL for cefpodoxime or minimum inhibitory concentration ≥2 µg/mL for ceftazidime, or ceftriaxone as a result of the bacterial culture test (liquid microdilution method). This criterion corresponds to the screening criteria for ESBL-producing bacteria according to the Clinical and Laboratory Standards Institute M100-S25 [9].

The predictor variables were broadly classified into variables extracted from the structured tables and those related to the patient’s history extracted from the text data.

The variables extracted from the structured table were patient age at the time of specimen collection, sex, number of days between admission and specimen collection, admission type, previous location of the patient before arrival at the hospital, and specimen type collected for the bacterial culture test.

The medical history variables extracted from the discharge summary text were preprocessed using the procedure: (1) from the discharge summary, we extracted paragraphs beginning with the string “Past medical history,” “Past Medical History,” or “PAST MEDICAL HISTORY”; (2) QuickUMLS was applied to the extracted paragraphs to extract the CUI with a Jaccard similarity coefficient of 0.7 or higher; and (3) dummy variables with and without each CUI were used for medical history.

First, we constructed an L1-regularized logistic regression model using only the variables extracted from the structured table. Subsequently, we used the variables derived from the structured table and those related to medical history extracted from the text to build either an L1-regularized logistic regression model or a LightGBM model. We chose L1-regularized logistic regression and LightGBM to build our predictive model primarily for 2 reasons. First, logistic regression is a straightforward linear model used for binary classification, while LightGBM is a more complex model that can handle both linear and nonlinear patterns. This combination allows us to cover a broad range of data behaviors. Second, we used L1 regularization with logistic regression to help manage the model’s complexity by selecting important features. For reference, we constructed a model that judges all positive cases, one that judges all negative cases, and one that judges randomly according to the ratio of positive to negative cases.

The models were constructed and evaluated using stratified group 5-fold cross-validation with patient ID as a group variable. This method ensures an equal class distribution in each fold, while samples from the same patient are not split across different folds. The receiver operating characteristic area under the curve (ROC-AUC) and precision-recall curve area under the curve (PR-AUC) were used to evaluate the performance of each model. The ROC-AUC evaluates the model’s ability to distinguish between classes, whereas the PR-AUC focuses on the model’s performance in terms of precision and recall, which are particularly valuable when dealing with imbalanced data sets [10]. These were calculated for each fold, and the average of the values from each fold was used as the ROC-AUC and PR-AUC of that model.

The total number of patients registered in the MIMIC-3 database was 46,520, of which 4046 underwent bacterial culture tests, and K pneumoniae or E coli were detected. After excluding 432 patients without a summary text on discharge, the final study population consisted of 3614 patients. Of the specimens collected from this study’s population, 5272 specimens were positive for K pneumoniae or E coli, which were the targets of this model.

Patient characteristics are summarized in Table 1, which shows no notable differences in mean age between negative and positive ESBL screening patients, although there was a slightly higher proportion of older patients (n=1160, 27.3%) aged 80 years or older among negative patients. The sex distribution was slightly higher for females (n=2477, 58.4%) in the negative group and almost the same in the positive group. The average time from admission to specimen collection was 6.1 (SD 11.9) days for negative patients and 12.1 (SD 18.4) days for positive patients. Before arrival at the hospital, the previous location of the patient did not differ markedly between the negative and positive patients. Specimens tested for bacterial growth showed that negative patients had slightly more urine specimens and slightly fewer sputum specimens than positive patients. However, these differences were not statistically significant.

The ROC-AUC of the L1-penalized model constructed using variables extracted from a structured table was 0.646 and the PR-AUC was 0.307 (Table 2). When variables extracted from the structured table and past medical history variables extracted from the discharge summary were used to construct the L1-penalized model, the ROC-AUC and PR-AUC were 0.653 and 0.335, respectively (Table 2 and Figure 1). The ROC-AUC of the LightGBM model was 0.707 and the PR-AUC was 0.369 (Table 2 and Figure 2). In the reference random classification model, the ROC-AUC and PR-AUC were 0.501 and 0.275, respectively (Table 2). The model that predicted all cases as positive or negative had a ROC-AUC of 0.500 and an undefined PR-AUC (Table 2).

Several key variables related to patient attributes and medical history are highlighted in the feature importance data for the LightGBM model. Specifically, “age at admission” has an importance score of 262, emphasizing its significant predictive value in the model. The “days from hospitalization to specimen collection” feature also plays a critical role with a score of 245.2. Medical conditions such as end-stage renal disease (ESRD) and Pseudomonas infection are also pertinent, with ESRD assigned an importance of 64.4, although the specific score for Pseudomonas infection is not listed in the top features shown. The data further include morbid obesity and diabetes mellitus (DM), which are integral for understanding patient outcomes, though their importance scores need to be specified from the full list. Additional features such as “fright,” “secondary,” “trachy,” and “severed” are less prominent, with “fright” having an importance of 28.7, indicating its relatively lower but still notable influence on the model’s predictions.

Compared with the reference random classification model, we observed an improvement in both the ROC-AUC and PR-AUC of our model, built solely on predictor variables from the structured table data. The performance of the model was further enhanced by adding features extracted from text-based medical histories.

A systematic review reported that the ROC-AUCs of existing predictive models without text for drug-resistant bacteria ranged from 0.48 to 0.93 [1]. The performance of our model was neither outstanding nor disappointing; it was somewhere in between. Key features contributing to the model’s performance included age; number of days since hospital admission; and medical history of end-stage renal disease, Pseudomonas infection, obesity, and DM. These have been noted in previous studies as risk factors for ESBL-producing bacteria and other drug-resistant bacteria, and it seems reasonable to assume that they are important factors for predicting ESBL-producing bacterial infections [11-15].

While our study’s predictors are consistent with known risk factors for drug-resistant infections, the retrospective nature of our analysis means we might not have fully accounted for all biases inherent in such data. Therefore, our findings do not imply causal relationships and should be interpreted with caution, particularly in clinical applications.

Features from the discharge summary text improved prediction accuracy; however, the feature “fright,” which appears to be unimportant for predicting drug-resistant bacteria, also appeared among the important features. This may be because of the limitations of NER using QuickUMLS. That is, “fright” may have been incorrectly identified because it contains the string “right.” Patients with a long medical history are more likely to include common words such as “right” in their medical history, which may have increased the risk of ESBL-producing bacterial infection due to the long medical history, resulting in a higher feature importance. The accuracy of the history extraction by NER may have led to the extraction of invalid features and prevented the model from reaching a high level of performance.

Another concern is the frequent use of abbreviations in discharge summaries. For example, “DM” is an abbreviation for diabetes mellitus. QuickUMLS performs NER based on string similarity, so the abbreviation “DM” and the full term “diabetes mellitus” are the same, but the strings are not highly similar, making extraction difficult.

Furthermore, the extraction by QuickUMLS of words that might seem less meaningful at first glance, such as “secondary,” “trachy,” and “severed” is another challenge. Specifically, “secondary” is associated with “neoplasm metastasis” (CUI: C0027627), “trachy” with “tracheotomy procedure” (CUI: C0040591), and “severed” with “severing” (CUI: C1306232). However, these words were not always used in the discharge summaries to indicate their corresponding CUIs.

The data used in our study were obtained from MIMIC-3, a single-institution database, and the extraction of information outside the patient’s institution, such as medical history, was possible only through textual information. Medical histories were extracted from the discharge summary texts; however, information such as antibiotic usage history, which is a risk factor for ESBL, may not have been detailed in the discharge summary, leading to potential underextraction.

Additionally, while the MIMIC-III database is open and makes our study results more reproducible, the reliability of its data cannot be fully guaranteed. This means that although our model shows trends similar to previous studies, suggesting it has some validity, the possibility that the model could be invalid cannot be dismissed.

In summary, although feature extraction from the discharge summary texts using QuickUMLS improved the accuracy of predicting ESBL-producing bacterial infections, incomplete data and difficulties in extracting information from the text may have prevented us from extracting all the data required for ESBL prediction. These obstacles may have contributed to the suboptimal performance of the proposed model.

In conclusion, we constructed a model that predicts ESBL-producing bacterial infections with accuracy comparable to that of previous studies using the publicly available MIMIC-3 data set.

Because our model was constructed using open data and open NER technology, it exhibited a high level of transparency. We believe that this model serves as a valuable reference for future studies in this field.

By extracting information from the text, we enhanced the performance of our model. We posit that if we can extract data from the text with even higher precision, we may be able to further improve the performance of our model.

The advent of transformer-based models, such as BERT and GPT, has led to notable improvements in medical natural language processing tasks [2,16,17]. Given the rise in natural language processing techniques, we believe that further applications for the extraction of information from medical texts, such as those used in our study, are promising.