A 10-year retrospective cohort of 62,521 births from January 1, 2015, to May 13, 2025, at Kettering Health was used to obtain a convenience sample of 1734 delivery encounters. The population prevalence of postpartum blood transfusions at Kettering Health was reviewed over the 10-year period and found to be 2% to 2.7%. The sample size was estimated using a 2% population prevalence, a z score of 1.96, a 1% margin of error, and 95% CIs. The postpartum transfusion prevalence was also evaluated in 6-month intervals between January 1, 2015, and May 13, 2025, for population distribution. Vaginal delivery occurred in 65% of the population. Delivery mode, delivery indications, and transfusion rates were reviewed. The top 50 cesarean section delivery indications were grouped into 7 categories with medical staff leadership and obstetrician input. Vaginal delivery in structured data is labeled as an “unspecified” indication for a cesarean section. However, indications for cesarean section that are not detailed may also be listed as “unspecified” in the electronic medical record (EMR). Each cesarean indication and vaginal delivery group was reviewed for whether they received or did not receive a transfusion to determine distribution over the 10-year period for quality review. Population race, ethnicity, and age at the time of delivery were not used in model training. Patient age at time of delivery was not calculated for the dataset due to concerns that the delivery date comprises part of an infant’s protected health information. Table 1 reports the characteristics of the total delivery population, and Table 2 reports the characteristics of the study cohort. Labeling of data was performed using existing, structured EMR data (Epic; Epic Systems) and a manual chart review by a physician. Patients with prior deliveries, prior transfusions, and prior cesarean sections were present in the convenience sample. Multiple births for the same patient’s delivery date were normalized. Breech presentation was excluded from the convenience dataset as the primary indication for cesarean delivery.

Structured data were taken from the EMR. Delivery mode was a binary variable with absence of a cesarean section labeled as a vaginal delivery. Delivery indications were grouped into 7 categories: elective, failed induction of labor, macrosomia, medical or surgical indication, other, repeat, and vaginal delivery. Cesarean indications that were unspecified in the structured data were vaginal deliveries or not labeled in the structured data. Missing data for having received a transfusion (n=2) or induction were assumed to be not present. This was confirmed in a manual chart review. Four cases of “elective” cesarean section in the structured dataset were corrected after quality review by the delivering health care provider. In the existing, structured dataset, “unspecified” indications for cesarean section (n=120) were not labeled. Manual chart review resolved unlabeled cesarean indications in the physician-reviewed dataset (n=120). If the clinical notes listed more than one indication, the category was labeled as “other.” The datasets were normalized for multiple gestations for a single patient delivery encounter. All personal health information, dates, and demographic information were removed from the datasets. Having received a transfusion was selected as the target outcome. A transfusion was defined by one or more units of blood given.

A generalized linear model, random forest model, and XGBoost model were evaluated for performance. The XGBoost model was chosen due to its performance, internal cross-validation, and feature importance reporting. One-hot encoding was used to convert categorical variables. Cesarean section and use of induction were binary columns. The transfusion units column, which was numerical, was converted to a churn column. The transfusion units column was removed from the datasets prior to XGBoost model training to prevent data leakage. XGBoost was used in R (version 4.4.0; R Project for Statistical Computing). The model was trained with data split into 80% training and 20% testing for both datasets. Class imbalance was addressed using scale-positive weight and tune grid parameters in the caret package. Internal validation was performed with k-fold cross validation set to 5. Hyperparameter grid tuning and adjustment of maximum depth were performed. Models were compared for performance. The Youden J statistic threshold (sensitivity + specificity – 1) was calculated for predictions. The best-performing clinically relevant model was used in a prototype clinical decision support app for alpha testing (details in Multimedia Appendix 1). The TRIPOD +AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Artificial Intelligence) checklist was used as a framework for model development (Multimedia Appendix 2 [16]).

This study was reviewed by the Kettering Health Institutional Review Board and determined to be exempt from human subjects research requirements.

Our model demonstrated that XGBoost was an accurate, clinically relevant predictor of transfusion (AUC=0.705; PR-ROC=0.78; and F₁=0.809). This model had similar performance to the existing structured dataset (AUC=0.71, PR-ROC=0.82, and F₁=0.80). The PR-ROC is a better metric for imbalanced datasets. Although the structured dataset had a higher PR-ROC of 0.82, the predictions and feature importance ranking were not clinically accurate. Model predictions for the structured dataset and the physician-reviewed dataset were comparable in performance for recall (0.97 vs 0.96), specificity (0.22 vs 0.25), and positive predictive value (0.69 for both). Accuracy was improved using the clinician-reviewed dataset (0.69 vs 0.708). However, a significant difference was noted in feature importance ranking and clinical predictions made with the clinical decision support app. The labeling of both cesarean sections and vaginal deliveries as “unspecified” indications for cesarean section affected the ranking of feature importance for the existing dataset. Thus, the dataset with existing EMR structured data did not give relevant clinical information to support a clinical decision tool. Explanation of model predictions and features are important for clinical support tools. Model explainability in this study was aided by interactive user interface outputs for low, moderate, or high risk stratification based on the threshold determined with the Youden J statistic (sensitivity + specificity – 1) [17]. (Figure 2; Multimedia Appendix 1) Researchers can upload external datasets for validation in the app. This feature allows predictions to be downloaded for review (Figure 3; Multimedia Appendix 3).

The XGBoost model was chosen for this study for its internal cross-validation, hyperparameter tuning, and feature importance ranking. Both datasets were evaluated with the same k-fold cross-validation (k=5). Hyperparameter tuning maximum depth was adjusted to improve model performance on the physician-reviewed dataset.

Model performance was compared for accuracy between an existing, structured dataset extracted from the EMR and a physician-reviewed dataset. Communication of transfusion rates and cesarean indication grouping prior to model training was an important aspect of data collection and feature selection. Physician engagement with data collection and labeling may help to explain the model’s predictions for future clinical use. Physician communication of the transfusion prevalence in our population and grouping of cesarean section indications led to development of the app. The physicians requested a clinical support tool that would help them discuss transfusion risk and delivery mode and fit into their clinical workflow.

In our study, delivery mode and medical or surgical indications for a cesarean section were important features. Delivery mode was one of the most predictive variables for PPH in the gradient boosting prediction model by Ahmadzia et al [7]. Despite an AUC of 0.83, the authors noted the model had difficulty predicting transfusion risk alone in a population with a 2.8% transfusion rate. Predicting transfusion risk in low-risk patients, however, remains challenging [18,19]. Many machine learning studies have sought to identify high-risk patients during labor. These studies have demonstrated machine learning to be useful in identifying high-risk populations compared to traditional risk-assessment tools [6,9,10,12,19].

Our clinical app was developed by physician request to aid clinical discussions of transfusion risk with a low, moderate, and high risk stratification. In this study, risk stratification was based on the Youden J statistic threshold (sensitivity + specificity – 1). This offers the clinician the ability to see how the model determines the classification. This display may improve patient and health care provider acceptance of the prediction. Physician input regarding the labeling and grouping of cesarean indications was important for data collection. Despite the promise of machine learning in PPH prediction, models designed for clinical use have yet to demonstrate utility in clinical practice. Clinicians use other machine learning–trained models to predict the success of a vaginal delivery after a cesarean section [20]. Thus, physicians may accept a clinical app to supplement discussions of PPH risk. It remains to be determined whether the model or app will generalize to other populations, improve patient outcomes, or be accepted by physicians and patients. Our model-supported app is a proof-of-concept approach to predicting PPH transfusion risk to facilitate shared decision-making. External validation is strongly recommended prior to app use in clinical settings. In addition, we recommend engaging physicians to participate in feature labeling to determine what features would be clinically meaningful to mitigate risk.

In our study, delivery mode followed by medical or surgical indications for a cesarean section were important features. As noted above, delivery mode was found to be the most predictive feature of PPH requiring a transfusion in the study by Ahmadzia et al [7]. Other studies have noted a 2% transfusion prevalence in their populations [3,12]. The low prevalence of transfusion in the study population and manual clinical data review pose resource challenges. For this reason, many studies use retrospective analysis of data to train models. Woo et al [11] found that large language models (LLMs) trained on clinical notes outperformed LLMs using structured data alone [11]. Clinically accurate information derived from LLMs may improve data quality, model development, and performance for future studies. LLMs trained on clinical notes may be a solution to obtain clinically accurate data in limited-resource and community settings.

Known risk factors for PPH were included in this study as indications for a cesarean delivery. High-risk conditions such as placental spectrum disorders, abruptio placenta, placenta previa, and HELLP (hemolysis, elevated liver enzymes, and low platelet count) syndrome were included in the medical or surgical indications for delivery. Prior PPH, grand multiparity, and prior cesarean section are high-risk conditions for PPH that would be known to the health care provider at the time of counseling for transfusion risk. In addition, risk factors that are unmodifiable prior to delivery such as age, BMI, race, and ethnicity were not included, as they would be known to the health care provider during counseling. Many studies have demonstrated machine learning models that can accurately predict PPH in high-risk conditions [6,8-10,12,18,21]. However, predicting PPH in low-risk patients remains challenging [9,18,19].

Limitations of this study include the retrospective nature of the review of records at a single community health system, the small sample size, the overlap of delivery indications, and the inability to demonstrate generalizability of the model with external validation.

Further study is needed to determine if our proof-of-concept machine learning model–based clinical app will be generalizable and meaningful in clinical care settings. The external validation of our model is beyond the scope of this study.

Our XGBoost model demonstrated accurate, clinically relevant prediction of transfusion (AUC=0.71, PR-ROC=0.78, F₁ score=0.80) on structured data. The model performance on a physician-reviewed dataset was comparable (AUC=0.705, PR-ROC=0.78, and F₁=0.809). Although these are similar metrics, the model gave more clinically appropriate predictions and feature importance ranking for the physician-reviewed dataset. For this reason, this model was selected for our proof-of-concept clinical decision support app. External validation is needed to see if this model will generalize to other populations. Further study is needed to determine if this app will be acceptable to patients and physicians as a useful prediction tool to improve patient care.