Fast Healthcare Interoperability Resources–Based Support System for Predicting Delivery Type: Model Development and Evaluation Study

Introduction

Background

The ability to provide care to both women and newborns during delivery is one of the most important aspects of health care and is often used as a metric to assess health care across different countries. Cesarean delivery is one of the most important aspects of delivering babies since it has a considerable impact on the mother’s health and well-being. Despite the increased prevalence of this procedure over the last few years, the reasons behind this trend still remain unclear. Reports suggest that this increment is a global phenomenon, with the rate of cesarean deliveries almost tripling from 6.7% to 19.1% between 1990 and 2014 [1,2]. Research on the impacts of cesarean deliveries has focused on the risk of infection, hemorrhage, organ injury, and complications related to anesthesia or blood transfusion [3,4]. There is also a higher risk of complications in subsequent pregnancies, such as uterine rupture, abnormal placental implantation, and the need for hysterectomy [5,6]. As for the infant, cesarean deliveries can lead to respiratory problems, asthma, and childhood obesity [5]. In light of this, in 2015, the World Health Organization stated that cesarean delivery rates higher than 10% were not associated with a reduction in maternal or newborn mortality, even though other complications could not be fully assessed [7]. In contrast, there is no evidence of the benefits of this procedure for women or babies when there is no clear medical need; therefore, it is paramount to focus on identifying and reducing such cases [2]. It was estimated that in 2018, there were 8.8 million unnecessary cesarean deliveries [8]. It was with this in mind that a committee was established in Portugal with the specific purpose of decreasing the percentage of cesarean deliveries nationwide. One of the policies resulting from this committee’s work was the reduction of government funding per inpatient cesarean delivery episode for hospitals with rates of cesarean deliveries above 25%; as of 2020, the number of cesarean deliveries in Portugal stands at approximately 36.3%, nearing the all-time high of 36.9% in 2009 [9]. Furthermore, studies have shown that several countries could benefit from similar policies [8]. A quantitative analysis estimated that a reduction in cesarean deliveries could save millions of dollars [10] worldwide. Therefore, lowering the proportion of cesarean deliveries can yield health and financial benefits for both institutions and patients alike. With these considerations in mind, we developed a machine learning, algorithm-based support system to assist clinical teams in identifying cases of potentially unnecessary cesarean deliveries. As such, in this paper, we propose to (1) elaborate on how clinical decision support systems for cesarean deliveries can be developed using interoperability standards; (2) understand, based on the data collected, which features have the most significant impact on predicting delivery type; (3) conduct a concise economic analysis to assess the potential financial impact of implementing the proposed clinical decision support tool; and (4) compare the system’s output with clinicians’ responses.

Rationale and Related Work

Regarding the related work, several teams already tackled the potential of predicting the delivery type before birth. We found studies related to predicting a successful vaginal birth after a previous cesarean delivery, such as the work of Lipschuetz et al [11], where a gradient boosting method was used to predict such an event using prenatal data to do so. Grobman et al [12] performed a similar study with a multivariable logistic regression model. Different modalities of data were also used to predict delivery type. Fergus et al [13] introduced a method of predicting delivery type using fetal heart rate signals. Similarly, the work from Saleem et al [14] proposed a method for predicting delivery type using interactions between the fetal heart rate and maternal uterine contraction. Finally, some studies focus on predicting the delivery mode, such as the work of Ullah et al [15], where a boosting algorithm was used to predict a delivery mode with enriched data sets. In addition, Gimovsky et al [16] introduced decision trees to predict cesarean deliveries by physician group with an area under the receiver operating characteristic curve (AUROC) of 0.73. The works of Rossi et al [17] resulted in a 7-variable model with an AUROC of 0.78, and the works of Guedalia et al [18] resulted in a model with an AUROC of 0.82, reaching 0.93 with a first cervical examination. Finally, the works of Meyer et al [19] focused on selecting something suitable for a trial of labor after cesarean delivery with an area under the precision recall curve graph around 0.351. However, to the best of our knowledge, there was no model tested in clinical practice with an interoperable format of communication such as Fast Healthcare Interoperability Resources (FHIR), which tried to not only predict delivery type but also provide support about possibly wrong deliveries, and none with simulation about financial implication, making our paper a potential novelty on different dimensions.

Methods

Materials

Data were retrospectively collected from 9 different public Portuguese hospitals across the country, focusing on obstetric information and encompassing maternal data, various fetal data points, and the method of delivery retrospectively. The inclusion criteria are all mothers with a registered outcome of the pregnancy from 2019 to 2020. There were no exclusion criteria. Each institution used identical electronic health record software, ensuring that the data columns remained consistent.

Clinical Comparison

The clinical comparison was performed by sending questionnaires to clinicians with a relationship with obstetrics to assess 10 patients, with only access to the variables used by the model, and to answer 3 questions for each. The first question was to give a score from 1 to 10 of how likely it was that a patient would give birth through cesarean delivery, the second question was to select the feature or variable that most influenced the decision, and the third question was to select which feature they would require to make a better assessment. We sent the questionnaire to 20 people and obtained 6 answers, totaling 60 patient assessments. For these 10 patients, we also predicted the delivery type using our model to compare it with the clinicians’ answers. These patients were new and were not seen by the model during the training phase.

Analysis

All null representations were standardized. Data were prepossessed by removing features with high missing rates (>90% overall). The imputation process was performed using the k-nearest neighbor imputation method (for continuous variables) or a new category (NULLIMP) for categorical variables. Weight was categorized into percentiles defined specifically for Portuguese babies [20]. For this study, the birth type was reduced to binary. All assisted birth were merged into vaginal birth, and cesarean delivery remained as the other class. Procedures and diagnoses were also used and were encoded as binary features, and we took the time to analyze each one of them to avoid leakage because there were procedures obviously related to cesarean deliveries and vaginal deliveries. Feature creation was performed through the free-text variable related to the prescribed medication. Medicine names were collected from it and converted into Anatomical Therapeutic Chemical classification group level 4, which represents chemical subgroups. We also created some new features from data in the data set, namely new categories related to the labor and condition of the baby. In addition, data quality issues were addressed, such as impossible values that were transformed into null values. The main variables affected by data quality were BMI or weight and gestational age. The data were split into training and test sets in a 0.75:0.25 manner. From the overall data sets that comprised over 200 columns, only a few columns were selected (see Table 1). We used a mixture of features selected by surveying the literature [21-23] and features with a high correlation with the outcome. The tested models were logistic regression, decision tree, random forest, 3 different boosting methods (as implemented by extreme gradient boosting [XGBoost], light gradient-boosting machine [LightGBM], and scikit learn), and a linear model based on stochastic gradient descent.

The evaluation was performed with repeated stratified cross-validation with 10 splits and 2 repetitions, with 2 full cycles of dividing the training set into 10 equal parts and using 9 as the training set and 1 as the validation set. The results are shown in Table 2. The application programming interface (API) for serving the prediction model was developed using FastAPI. We wrote all the code in Python (version 3.9.7; Python Software Foundation).

Ethical Considerations

This study received institutional review board approval from all hospitals included in this study with the following references: Centro Hospitalar São João (08/2021), Centro Hospitalar Baixo Vouga (12-03-2021), Unidade Local de Saúde de Matosinho (39/CES/JAS), Hospital da Senhora da Oliveira (85/2020), Centro Hosptilar Tamega Sousa (43/2020), Centro Hospitalar Vila Nova de Gaia/Espinho (192/2020), Centro Hospitalar entre Douro e Vouga (CA-371/2020-0t_MP/CC), and Unidade Local de Saúde do Alto Minho (11/2021). All methods were carried out per relevant guidelines and regulations. The need for informed consent was waived by the ethics committee.

Results

Descriptive Statistics

The number of samples varied across the hospitals, ranging from 2364 to 18,177. Distributions of the selected variables are presented in Table 1. The sum of all samples was 73,351. The outcome variable distribution is stated in Table 3.

Model

The AUROC is presented in Table 2 for the best hyper-parameters found for each algorithm in the training data. All models used the variables indicated in Table 1.

While XGBoost was the best-performing algorithm, we selected LightGBM [24] because of its speed and lower memory requirements, which we believe are better suited for deployment in a low-hardware environment. The threshold selected for deploying the model was 0.7457, which rendered the metrics in the test set, as shown in Table 4.

Deployment

The purpose of this model is to serve as an API for usage within a health care institution and to act as a supplementary clinical decision support tool for obstetrics teams. For this to happen, a health information system must make the requests to the API. Even though a concrete, vendor-specific information model and input health information system were used, we hope to create a more interoperable clinical decision support system that can be used by every system that acts on birth and obstetrics departments. Therefore, we built it around the HL7 FHIR standard (R5 version) to simplify the method of interacting with the API. This decision, opposed as to using a proprietary model for the data, sits upon the usage of FHIR resources: bundle and observation for request and returning the result as a message through a custom operation called “$predict.” It is intended to publish the profiles of these objects to facilitate access to the API using standardized mechanisms and data models. The current build of the profiles can be seen in the published FHIR implementation guide where the current specifications are described in detail [25]. The process is illustrated in Figure 1. We deployed this model in production in a single hospital without a user interface, collecting only the data and predictions for later discussion and analysis. We collected 3231 requests. During this period, 123 (3.8%) alarms were triggered. From this, we tried to understand the level of certainty for the decision and check the difference from the threshold of these alarms. The distance to the threshold for 73 was lower than 0.1 and was bigger than 0.1 for 50 (1.55%) cases.

‎

Clinical Comparison

The median scores given by each clinician are presented in Figure 2. We also predicted the result using our model as stated in Figure 2. The model misclassified only 1 record (ID 4). As for the analysis of missing features for the responders, they were divided into 3 categories: (1) existent in the data set but not included in the model, (2) nonexistent in the data set, and (3) existent in the data set and included but that particular information was not filled for the patient assessed. Out of 60 responders, this rendered a total of 37 (62%) with nonexistent features and 23 (38%) with existent features but no information was provided at that moment. No feature mentioned existed in the data set but had not been included in the model. From the 37 nonexistent features, 14 (38%) were new clinical assessments, 14 (38%) were linked to information from previous births, 5 (15%) were connected to more in-depth information about provided information (ie, a motive for induction), and 4 (11%) were related to the mother’s choice (if she wanted a cesarean delivery). As for feature importance, from the 60 answers, 33 (55%) stated that labor was the most important factor. Further, 9 (15%) stated the number of previous vaginal births, 5 (8%) stated the evolution of weight, and another 5 (8%) stated the number of previous cesarean deliveries as being the most important. The remaining 8 (14%) were various features, such as BMI, neuroaxis techniques, gestational age, and weight of the mother. Of all of these, 54 (90%) were in the top 10 features of the model.

‎

Potential Financial Impact

The financial support provided to public hospitals in Portugal is partially tied to the rate of cesarean deliveries. To assess the potential impact of this mechanism on Portuguese public hospitals, we conducted a simulation. We got data for every public hospital for the last 12 months and applied a 3.8% reduction (the rate of warnings triggered in the new data set) and recalculated the rate of cesarean deliveries. The increase in support was calculated by the state-mandated rate as shown in Table 5. With this new rate, we observed that implementing our tool would result in financial benefits for 30% (15/48) of the public hospitals. Specifically, 5 hospitals would begin receiving support instead of no support at all. Further, 3 hospitals would experience a doubling of their financial benefit, while 2 hospitals would see a 50% increase. Furthermore, 1 hospital would receive an additional one-third of financial support. If we assumed that only half of the warnings found in the new data were actually true (1.9%), we found that only 6 hospitals would be benefited: 3 from 0 to 0.25, 2 from 0.25 to 0.50, and 1 from 0.50 to 0.75.

Discussion

Principal Findings

The first thing to address about this model is the number of biases that we introduced in the model by choice. We joined all vaginal delivery types into a single category (assisted and nonassisted), which introduces a bias since these delivery modes are indeed different. Second, the fact that we want to predict if the delivery type was wrongly chosen, mainly for the case of a cesarean delivery that did not need to be so, is also a bias. We used this approach because the initially collected data did not have the representation of such events. Thus, the biases of possibly wrong delivery types were present in the training data. We attempted to minimize this issue by selecting a threshold that gave the model higher sensitivity than specificity so that only large probabilities would trigger an alarm for human consideration. Parallel to this, we are starting to gather labeled cases, with the help of clinicians to create a better training data set. Furthermore, since the data were collected from different hospitals, differences in the data input can also occur. Even though the health information system is the same, the processes that originate the data and are being used for secondary purposes could introduce several biases in the data. This is an issue that was accepted from the start regarding the mechanism of data collection and model training. Despite this, we reached a model with a very high AUROC (88%, 95% CI 0.8795-0.8815), which is encouraging when compared to the state-of-the-art, which is between 0.73 and 0.82 [16-18]. Moreover, assuming that more data are provided, and proper labeling is done regarding the outcome variable (such as a clinical evaluation of needless cesarean deliveries) is added as well, a better model could be developed.

Regarding the preliminary clinical evaluation, it was only possible to obtain an overview of the possible comparison due to the number of responders. Despite that, the results are encouraging, since the model seems to behave better than humans with the data provided. However, this is a biased vision since clinicians in the real world have access to more data and information than the model has. It is encouraging, but caution is advised before more tests and evaluations are done. As for the deployment, future work could be the improvement of the API to map all variables to an ontology such as SNOMED CT or similar, making it easier for every system and person to access it and obtain a suggestion of the delivery type. Finally, we believe the assessment can be improved. A more robust clinical assessment is necessary, as well as a thorough analysis of the impact of the tool in the real world, since we need to create the bridge between the results of the model and how clinical decisions are affected by it. A full cost-effectiveness analysis is also necessary to understand the real-world impact of the model. Further, 1 interesting result is the fact that 38% of the answers regarding the most important data element missing from the patient record refer to data that are being collected but was missing for that specific patient, raising an important question about data input methodology, interoperability, and quality. If we cannot have access to data when these matter the most, these can become meaningless. Missing data are a problem of biomedical data as a whole. However, when specifically targeted at machine learning usage of this data for predicting something, we did not find any work comparing them with clinicians. However, we did find reporting of similar missing values in obstetrics data [27] and we also found works of a similar nature using machine learning models with robust handling of missing data such as XGBoost [28] to counter this problem. This indicates that our model has the potential to counter the missing data problem as well since LightGBM can also handle missing data natively.

Conclusions

We believe we have developed a robust system capable of detecting potentially incorrect cesarean delivery decisions, which could positively impact real-world medical practice. However, before implementation, several challenges must be addressed, particularly the need for further evaluation of the system’s impact on clinical decision-making and the reasons underlying suboptimal delivery-type decisions. Cesarean deliveries may be performed for various reasons, from a mother’s preference to a decision made by the obstetrics team. This system is not designed to impede medical practice or to highlight flawed decisions, potentially scrutinizing specific professionals. Such caution is necessary when implementing systems like these. While having a high AUROC is beneficial, the real-world impact is another consideration. The assumptions and biases associated with autonomous systems supporting clinical practice must be carefully considered. Nonetheless, the metrics and results we have achieved so far are promising for positively influencing health and economic outcomes.