This retrospective study involved 2406 patients with critical illnesses admitted to medical ICUs, including 1 each at Dongguk University Ilsan Hospital (DUIH) and Kangwon National University Hospital (KNUH) between January 1, 2010, and February 28, 2022. We used the data collected from DUIH and KNUH as the internal and external validation data sets. The internal validation data set contained 977 patients, while the external validation data set contained 1429 patients.

Patients with critical illnesses (>18 years) with intravenous vancomycin treatment history and who had at least 1 test for vancomycin TDM were eligible for this study. Vancomycin TDM was investigated for trough-level information. If the TDM was checked multiple times, only the first TDM value was selected. However, in patients with normal renal function, if the interval between stopping vancomycin and readministering it was 2 weeks or more, it was considered an independent TDM value and used for analyses. Patients who received oral vancomycin or were aged 18 years or younger were excluded. The total serum vancomycin concentrations were determined using a fluorescence immunoassay (VANC3, Cobas c 702; Roche Diagnostics).

The data set was split into training and testing groups in a 9:1 ratio, with 879 and 98 cases representing the internal data set, respectively. For external validation, 1429 cases from KNUH were used. To evaluate the model’s generalization and the range of expected errors of the classifiers, 10-fold cross-validation was performed on the internal training data set, with a 9:1 ratio and 791 and 88 cases, respectively. The model was tested on both the internal and external test data sets for each fold and the metrics obtained were used to calculate the 95% CI over the 10 folds. The internal and external data sets remained constant across all folds. To evaluate the generalization ability of the joint multilayer perceptron (JointMLP) model, we used 2394 cases from the Medical Information Mart for Intensive Care–IV (MIMIC-IV) [17] as a US critical care database used for large-scale multi-institutional research for external validation.

We proposed a JointMLP to predict the vancomycin TDM level in patients in the ICU. The JointMLP model is an ensemble model, meaning it combines the predictions of multiple individual MLPs (multilayer perceptrons) to generate a final prediction. The MLP is an artificial neural network. The MLP consists of multiple layers of interconnected nodes (also known as neurons) that perform mathematical operations on the input data to generate an output. In the case of the JointMLP model proposed for vancomycin TDM prediction, each MLP in the ensemble consists of 3 hidden layers with a hidden unit size of 32, and the LeakyReLU activation function is used to introduce nonlinearity into the model (Figure 1). Using an ensemble helps reduce the risk of overfitting and improves the accuracy and robustness of the model. In this case, the JointMLP model comprises 100 different MLPs, each with its own set of weights learned during the training process. To find the best set of hyperparameters for the JointMLP model, grid search is used, which systematically tests different combinations of hyperparameters to find the combination that yields the best performance on a validation data set. In this case, the tuned hyperparameters included the number of MLPs, the number of hidden layers, and the size of the hidden units. In a JointMLP model, the MLPs are trained independently in parallel. During training, each MLP in the ensemble receives a randomly selected subset of the training data and learns to make predictions based on that subset. The individual MLPs are then combined to generate a final prediction, typically by taking the average of their outputs or using a weighted combination. The MLPs are trained independently of each other and in parallel. Our JointMLP model is available on the websites [18] for predicting TDM levels and recommending the frequency and dose.

We compared 4 other models: the PPK model [19], XGBoost, TabNet, and 300-layer MLP. The PPK is a pharmacokinetic model used to assess the relationship between drug concentration and time in the body. XGBoost is an open-source library for decision tree–based gradient-boosting machine learning. The XGBoost model especially works well on tabular data learning. It is a vital model for distributed training or regularization. TabNet is a deep learning model specialized for tabular data set analysis. TabNet could be optimized by automatically transforming input variables and using additional validation data to select the required variables.

For XGBoost, the hyperparameters are eta (learning rate), gamma (minimum loss reduction required to make a further partition on a leaf node of the tree), and max_depth (maximum depth of a tree). The grid search was performed to find the best combination of these hyperparameters. The eta, gamma, and max depth values were 0.2, 0.001, and 6, respectively.

For TabNet, the hyperparameters are n_d (width of the decision prediction layer), n_a (width of the attention embedding for each mask), n_steps (number of steps in the architecture), gamma (coefficient for feature reusage in the masks), and verbose (verbosity for notebook plots). The grid search was performed to find the best combination of these hyperparameters. The values chosen for n_d, n_a, n_steps, gamma, and verbose were 8, 8, 3, 1.3, and 0, respectively.

The 300-layer MLP model was constructed to verify the performance of the JointMLP model. The hyperparameters for the MLP model were chosen to have the same depth as the JointMLP model, which is 100 horizontal MLPs times 3 hidden layers. The number of hidden layers was set to 300, and the number of hidden units was set to 32.

Variables that are associated with vancomycin PK/PD were selected. These variables included the total dose of vancomycin from start to end, the usage of the loading dose at the first use, the total number of vancomycin infusions, the dose of vancomycin per infusion, the mean infusing interval of vancomycin, the interval from the start of vancomycin administration to the measurement of vancomycin serum levels, age, sex, height, body weight, serum creatinine levels, serum vancomycin levels, dialysis, and the total volume of transfusion from the start of vancomycin administration to the measurement of vancomycin levels. Patients with impaired kidney function may have issues with the metabolism and elimination of vancomycin. The estimated glomerular filtration rate (eGFR), which represents renal function, was calculated using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) formula and used as a variable. For the normalization of the data, for both the input and target variables, all values were scaled in the range (−1 to 1) using the minimum-maximum scaler. Moreover, we only had binary categorical variables in the model. Therefore, no special categorical variable encoding was performed in the model but a simple change to 0 and 1. To select significant input variables, we grouped variables into 5 categories. Each variable was classified as (1) baseline demographics—age, sex, body weight, and height; (2) drug administration-related variables; (3) variables related to the volume of distribution; (4) eGFRs using serum creatinine levels; and (5) variables related to the amount of transfusion. With the above-grouped categories, first, we have selected the default variables that would be used throughout all scenarios: the baseline demographics and the drug administration–related variables. Then, from the remaining 3 categories, with the clinician’s consultation, we tried to minimize the redundancy of the variables that would have the same meaning. Moreover, in accordance with the feature importance of each variable, we were able to find the best combination of model input variable sets. Furthermore, the contribution of the input data to the model in the test set was determined through the Shapley value of Shapley Additive exPlanations (SHAP) [19]. A high mean absolute estimated Shapley value indicates that the variable has a stronger impact on the output value.

The primary end point was the predictive performance of the vancomycin level within the therapeutic range. The baseline variables and patient characteristics of internal and external data sets were presented as frequencies with percentages or mean values with SDs. Between–data set comparisons were performed using the paired 2-tailed t test for continuous variables or the chi-square test for categorical variables. The measured serum vancomycin levels were used as true values. The predictive abilities of 4 kinds of models for vancomycin trough concentrations, using the PPK model on the website of the infectious diseases management program at the University of California, San Francisco [20], machine learning models (XGBoost) [21], and deep learning models (TabNet [22] and JointMLP), were evaluated for bias and precision by calculating the mean absolute error (MAE), root-mean-square error (RMSE), R², and adjusted R². The paired t test on MAE was used [23,24] to confirm significant differences in prediction performance between models. P values of <.05 were considered statistically significant. All analyses were performed with R software (version 3.4.4; R Foundation for Statistical Computing) and Python (version 3.8.12; The Python Software Foundation) using the SciPy 1.9.1, PyTorch 1.10.2, and PyTorch Lightning 1.6.4 libraries.

This study involved a retrospective analysis of medical records and did not involve collecting prospectively obtained data. As such, the study was considered to pose minimal risks and did not directly affect patients’ rights, welfare, or clinical care while offering recognized social benefits. To protect patient confidentiality during data collection and record review, all investigative records were securely stored in an encrypted database, and published results did not include any participants’ names or identifiers. The study received ethical approval from the institutional review board of Dongguk University Hospital (DUIH 2021-08-015) and Kangwon National University Hospital’s Institutional Review Board (KNUH-2022-03-020). This study was performed per the principles of the Declaration of Helsinki.

Out of the 977 patients included in the DUIH data set, the testing data set was composed of 98 (10%) patients, with the other 879 patients belonging to the training data set. All 1429 patients included in KNUH were for external validation. The baseline characteristics did not differ significantly between the testing and training groups (Table 1). However, there were more older patients and shorter heights in the external validation set. The estimated renal function by the CKD-EPI method was worse in the patients for external validation. The patients included in MIMIC-IV showed significant differences in sex distribution compared to internal data sets or patients in other ICUs in Korea, and the proportion of non-Hispanic White patients was over 66.4% (n=1591).

Compared to the PPK model, the XGBoost model only significantly improved the predictive performance of external data sets (20.43 vs 11.59, a 76% change by RMSE and 11.87 vs 8.75, a 36% change by MAE; Table 2) when evaluated by RMSE and MAE. Although statistical significance could not be determined, the predictive power of internal data sets also improved by 8% (10.38 vs 9.58 by RMSE) and 16% (6.68 vs 5.75 by MAE).

The TabNet model showed significantly better predictive performance for vancomycin TDM after vancomycin treatment than the PPK model in external data sets (20.43 vs 13.89, a 47% change by RMSE and 11.87 vs 7.50, a 58% change by MAE; P<.001, respectively). Contrary to the prediction improvements compared to the PPK model in the external data set (TabNet 47% vs XGBoost 76% by RMSE), there was no significant difference between TabNet and XGBoost in predictive power for vancomycin TDM on the internal data set.

In both internal and external data sets, the JointMLP model had better predictive performance for vancomycin TDM among the models we compared. It improved the predictive power up to 31% (6.68 vs 5.11) based on MAE for the vancomycin concentration after vancomycin infusion on the internal data set compared to PPK. Additionally, our JointMLP model was more accurate (11.87 vs 6.56, 81%) in predicting the vancomycin levels on the external data set than the PPK method. The proposed JointMLP model significantly outperformed XGBoost and TabNet with a 13% (5.75 vs 5.11) and 14% (5.85 vs 5.11) improvement in predictive accuracy under MAE on the internal data set, respectively. Additionally, our trained JointMLP showed better performance not only on the internal data set but also on the external data set that consisted of 1429 patients with critical illnesses by 34% (8.75 vs 6.56) and 14% (7.50 vs 6.56) when compared with XGBoost and TabNet. Furthermore, JointMLP’s RMSE performance is higher than that of other models on all data sets, which means that the proposed model is more robust than theirs against outlier data.

Within the external data set, we clearly see that all models are statistically significant among themselves (Figure 2). However, the P value of JointMLP within the internal data set expressed a trend more robust than all other models while showing statistical significance only when compared with PPK with MAE. The mean error and variance of the JointMLP model were nearer to zero and smaller in both internal and external data sets than in the PPK model (Figure 3).

The most influential variables in TDM prediction were eGFR CKD-EPI, the average dose of vancomycin, and the average volume of distribution in both internal and external data sets. These variables consistently had high SHAP values in both data sets (Table 3).

The purpose of Table 2 was to compare the predictive performances of the 300-layer MLP model and the proposed JointMLP model. The results showed that JointMLP outperformed the 300-layer MLP model in both the internal (8.27 vs 9.94, a 23% improvement by RMSE, and 6.98 vs 5.11, a 37% improvement by MAE) and external (9.50 vs 9.94, a 5% improvement by RMSE, and 7.45 vs 6.56, a 14% improvement by MAE) data sets, indicating the effectiveness of the boosting ensemble approach used in JointMLP. Figure 2 shows the P value analysis of the comparison between JointMLP and the 300-layer MLP model. The analysis revealed that the difference between the 2 models was statistically significant (P<.001) within the external data set. However, compared to other models, the difference between JointMLP and the 300-layer MLP model was not statistically significant (P=.09) within the internal data set.

The findings presented in Table 4 clearly indicated that the JointMLP model performed better than both the PPK and 300-layer MLP models by a significant margin of 160% and 25%, respectively, when evaluated using the RMSE metric. This improvement in predictive performance was observed even in the MIMIC-IV data set, which included patients from multiple centers and races. The JointMLP model also demonstrated superior performance compared to the XGBoost and TabNet models, with improvements of 5% and 4%, respectively. When examining the MAE metric, the JointMLP model significantly outperformed the PPK model by 90%, with further improvements of 8%, 6%, and 5% over the XGBoost, 300-layer MLP, and TabNet models, respectively.

All models, except for the 300-layer MLP, were statistically significant in the MIMIC-IV data set (Figure 4A). The most influential variables in TDM prediction using the JointMLP model were consistent with eGFR CKD-EPI, average vancomycin dose, and average distribution volume, which were variables identified in the internal data set (Figure 4B). The 300-layer MLP was statistically significant (P<.001) compared to the PPK model. Additionally, the JointMLP model had a mean error and variance that were close to zero and smaller than those of the PPK model in the MIMIC-IV data set (Figure 4C and D).

The PPK method is widely used for predicting vancomycin TDM levels. However, it is population-dependent, leading to inappropriate results for some patients. Therefore, this study proposes a consistent decision-making system that is population-independent and unaffected by clinicians’ abilities by applying a joint MLP model that integrates various successful deep learning models and decision trees. The proposed JointMLP showed the best performance for predicting vancomycin trough concentration in all data sets compared to other models. Significantly, in the MAE metric, the JointMLP model improved performance by 90% over the PPK model. Results mean that the proposed model can be applied to predict appropriate vancomycin trough concentrations in patients with critical illnesses who require vancomycin treatment in various situations.

In our study, we compared the performance of 3 machine learning models for predicting vancomycin levels: JointMLP, XGBoost, and TabNet. XGBoost is a popular gradient-boosting algorithm known for its high accuracy and flexibility, while TabNet is a newer deep-learning model [22] specifically designed for tabular data. Compared to XGBoost and TabNet, our JointMLP model outperformed both in terms of predictive accuracy. One of the main reasons for this is that our model was designed to handle noisy data samples, a common characteristic of medical data sets. A deep neural network architecture accomplishes this with multiple layers, which allows the model to learn complex relationships between the input features and the output variable. In contrast, XGBoost and TabNet are both tree-based models, which may need to be more effective at handling noisy data.

Compared to previous studies [15,24-26], our proposed JointMLP model performs better in accurately predicting vancomycin levels. Specifically, our model outperforms XGBoost and TabNet, 2 well-known models in analyzing tabular data and predicting drug concentrations. Moreover, our JointMLP model is designed to handle noisy data samples effectively, a crucial feature for real-world applications in the medical domain where data quality can be highly variable. Our model’s robustness is demonstrated by comparing it to a 300-layer MLP, confirming the effectiveness of our model structure in handling vancomycin TDM predictions. Importantly, our model is trained on a larger and more diverse data set than previous studies and tested on an unseen data set, demonstrating its generalizability and suitability for predicting drug concentrations. Overall, our findings highlight the potential of deep learning models in improving the accuracy and reliability of drug concentration prediction, which can lead to better treatment outcomes and improved patient health. Overall, our findings suggest that the JointMLP model is a promising approach for predicting vancomycin levels, particularly in the presence of noisy data. However, the choice of which model to use may depend on the specific characteristics of the data set being analyzed and the goals of the analysis.

When considering implementing our approach in clinical settings, there are several factors to consider. First, although we extended our model to validate across diverse ethnic groups by including patients from Asia and other regions in the MIMIC-IV data set, we acknowledge that there is still room for improvement in this regard. We used a larger data set than previous studies, with 977 samples as the internal data set and 1429 samples as the external data set. Additionally, we sought to develop a generalized model by applying the MIMIC-IV data set, which includes 2394 samples. However, additional validation with external data sets is necessary to enhance further the generalizability of our model for predicting vancomycin levels. Second, our comparison of the predictive performance with the PPK method was based on the trough-based target method, which is easily used in the treatment of patients with critical illnesses, rather than the AUC/MIC-specific range that is currently recommended. Therefore, further research is needed to compare the AUC/MIC range with our approach.

Despite these limitations, our deep learning model, if implemented in actual clinical practice, could significantly improve the treatment outcomes of ICUs by supporting clinical decision-making in a more standardized and consistent manner. Our proposed JointMLP model will continue to evolve and update its performance to mimic the human brain better and determine the optimal vancomycin dose.

In this study, proposing the JointMLP approach in clinical settings has significant implications for public health, as it can help optimize treatment outcomes for patients with critical illnesses in ICUs. By providing a more accurate and consistent method for predicting vancomycin levels, the model can reduce adverse events associated with suboptimal vancomycin dosings, such as the increased risk of bacterial resistance, longer hospital stays, and higher health care costs. Furthermore, the ongoing development and improvement of the JointMLP model through continuous learning and updating can lead to more effective treatments and better health outcomes for patients, improving the accuracy and reliability of other clinical prediction models in the future. Overall, the implementation of the JointMLP approach has the potential to improve public health outcomes and benefit patients worldwide.