The N3C offers the most extensive harmonized repository of COVID-19–related patient data in the United States, integrating EHRs from more than 70 health care institutions nationwide [35,36]. This centralized initiative provides a unified data platform through the N3C Data Enclave, enabling researchers to conduct large-scale, high-quality analyses focused on COVID-19 outcomes, treatment effectiveness, and health care disparities [36,37]. This study used the limited dataset (LDS) available through the N3C, which contains individual-level patient data with direct identifiers removed. This approach maintains patients’ privacy while preserving the detailed clinical information necessary for research [38]. The dataset conforms to the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), which standardizes clinical data structures across contributing institutions to support consistency and reproducibility [36,39]. For inpatient data analysis, the N3C employs the concept of a “Macro-visit,” encompassing hospital admissions, observation periods, extended multiday stays following outpatient procedures, emergency department visits, and overlapping outpatient or telehealth encounters that occurred during the COVID-19 pandemic [27,40].

The primary objective of this research is to predict outcomes for patients hospitalized with COVID-19, with a specific focus on in-hospital mortality. The unit of analysis is the patient hospitalization. To support this objective, the data must be transformed so that each hospital stay is represented by a single, consolidated record—commonly referred to as a flat table or analytic file—which is a typical structure for applying ML methods.

Data preprocessing contains other decisions that were not investigated for the simplicity of the presented work.

After decision C, data were filtered to include only patients whose first positive COVID-19 diagnosis occurred between August 1, 2020, and December 31, 2021. This period was selected to avoid the earliest phase of the COVID-19 pandemic, when testing availability, diagnostic coding practices, and clinical management protocols were rapidly evolving [25]. Restricting the analysis to this timeframe provides a more consistent basis for cohort comparison while maintaining a large and diverse patient population. Similar periods have been used in prior N3C studies of COVID-19 outcomes [36]. Then, hospitalization data were merged with other core data to be used in the final analysis with a unique identifier and variables including gender, date of birth, race, ethnicity, and age at death. Patients younger than 18 years were excluded from all cohorts. The analysis was restricted to adults aged 18 years and older, consistent with the standard definition of adulthood used in clinical research and prior N3C studies of COVID-19 outcomes [36]. Pediatric patients differ from adults in disease presentation, hospitalization patterns, and clinical outcomes. Restricting the study population to adults therefore provides a more clinically comparable population for evaluating the impact of cohort construction decisions [36,39].

This experiment identifies how data processing decisions can impact model performance across demographic subgroups. Similar to Experiment 2, the same 3 analytic approaches—maximum AUC classification, direct AUC regression, and AUC gap analysis—are applied across the 3 models—LR, RF, and GB.

However, in this experiment, the analysis is performed within stratified demographic groups. The test data are stratified by sex (male and female), race (Black, White, and Asian), and ethnicity (Hispanic or non-Hispanic). For each subgroup, shifts in model performance due to different data processing decisions are examined. This helps to determine whether certain decisions disproportionately impact certain demographic groups. The experiment is designed to help identify potential sources of bias and guides efforts toward more equitable predictive modeling.

The study was reviewed by the George Mason University Institutional Review Board (IRB), which determined that the proposed activity is not research involving human subjects as defined by the US Department of Health and Human Services (DHHS) and Food and Drug Administration (FDA) regulations. Therefore, the need for ethics approval was waived. The IRB reference number is #1706572. This study complies with the principles of the Declaration of Helsinki. The need for informed consent to participate was waived by the George Mason University IRB, as the study does not involve research on human subjects according to the relevant regulations.

Consistent with the analytical framework outlined in the Methods, results are organized into 3 components reflecting key aspects of the study design. We first assess predictive performance across cohorts and demographic subpopulations to evaluate model accuracy and fairness. We then investigate the impact of data processing decisions on model performance across alternative cohort constructions. Finally, we examine the extent to which these effects vary across demographic subgroups. These findings collectively highlight the role of cohort selection and preprocessing decisions in shaping both predictive performance and equity.

Predictive performance across cohorts and demographic subpopulations demonstrated notable variability, reflecting the impact of cohort construction and data processing decisions on model outcomes. Across both cohort sets (Set 1 and Set 2), the models achieved moderate to high discriminative performance, with AUC values generally ranging from approximately 0.59 to 0.86 across different models and evaluation settings. However, performance was not consistent across demographic groups, with some subpopulations exhibiting wider variability in AUC ranges. These findings highlight the sensitivity of model performance to both cohort definition and population characteristics, underscoring the importance of evaluating predictive models across diverse groups.

It is important to note that for Native Hawaiian and multirace subgroups, some models show an AUC of zero due to the absence of data, leaving no cases for training or evaluation. Table 1 summarizes the AUC ranges for LR, RF, and GB models across Cohorts 1‐16 and 1‐64, based on the entire dataset and stratified by demographic groups.

Model performance varied substantially across cohort definitions, indicating that data processing decisions play a critical role in shaping predictive outcomes. The importance of specific decisions differed depending on the cohort set and modeling context, suggesting that no single configuration consistently yields optimal performance. To better understand these effects, we further analyze how individual data processing decisions influence model accuracy across multiple evaluation settings.

To further interpret the findings, results are compared across multiple dimensions: within a single cohort set, across different cohort sets, across different models, and across the analytical approaches themselves. Due to page limitations, results are presented for a subset of approaches as representative examples.

Model performance varied across demographic subgroups, indicating that the impact of data processing decisions is not uniform across populations. The importance of specific decisions differed by subgroup, suggesting potential disparities in model behavior and performance.

The impact of different decisions on cohort selection and cohort formation can be observed. These decisions affect subpopulations differently. Across different demographic groups, the quality of the model is influenced by different decisions to varying degrees. Certain decisions become more important depending on the subgroup, and the range of influential decisions varies across demographics. For illustration, a subset of representative plots is presented. Sex-specific differences (female and male) are provided in Multimedia Appendix 5, while Figure 7 presents differences by race (White and Asian) and Figure 8 presents differences by ethnicity (Hispanic and non-Hispanic).

Fairness analysis using demographic parity and equality of opportunity revealed substantial variation across cohorts and demographic subgroups [46-48]. While sex groups (female and male) exhibited relatively similar fairness patterns, notable disparities were observed across racial, ethnic, and age groups. In particular, certain racial categories and age groups showed consistently lower fairness metric values across multiple cohorts, indicating potential inequities in model performance (Figures S2 and S3 in Multimedia Appendix 5).

Because subgroup-level fairness metrics depend on both subgroup size and the distribution of outcome cases within each subgroup, these results should be interpreted cautiously. In this study, demographic parity was calculated as the proportion of predicted positive cases within each subgroup, while equality of opportunity and equalized odds were calculated among observed positive and negative cases, respectively. Therefore, when a subgroup contained very few patients, or when positive or negative outcome cases were absent, the corresponding fairness estimates could be unstable or not estimable. For this reason, fairness results for small demographic groups, including Native Hawaiian and multirace patients in some cohorts, should be interpreted as descriptive indicators of potential subgroup differences rather than definitive evidence of bias.

Calibration analysis further demonstrated variability in the alignment between predicted probabilities and observed outcomes. Across both Set 1 and Set 2, calibration curves showed that model reliability differs depending on cohort construction and training-testing combinations (Figures S4 and S5 in Multimedia Appendix 5). These findings highlight that models with similar AUC values may exhibit different levels of calibration and fairness, reinforcing the importance of multi-dimensional evaluation. Due to space limitations, representative calibration plots for Set 1 and Set 2 are presented in Figures S4 and S5 in Multimedia Appendix 5, while the complete set of calibration results is provided in Multimedia Appendices 6 and 7.

The study systematically explores the impact of cohort selection criteria on ML model performance and demographic disparities in predicting COVID-19 outcomes. Overall, the findings address the study objective by showing that model performance and subgroup results are sensitive to cohort construction decisions. The findings underscore the critical role of data processing decisions in shaping model accuracy. Notably, decision D (time inclusion/exclusion) was consistently associated with variation in model performance, particularly in Set 1, which comprises 16 cohorts based on 4 primary decisions.

The analysis reveals that different ML models—LR, RF, and GB—respond variably to the same data processing decisions, indicating that the perceived importance of these decisions is not consistent across models. This variability highlights the need for careful consideration of model choice in conjunction with cohort selection criteria to ensure robust and reliable predictions. These findings reinforce prior concerns in the literature that differences in data preprocessing—rather than model selection alone—can substantially influence observed performance, thereby affecting reproducibility and generalizability.

Furthermore, the study delves into the impact of data processing decisions on model performance across demographic subgroups, including gender, race, and ethnicity. The results indicate that certain decisions disproportionately affect specific demographic groups, potentially introducing bias and exacerbating health disparities. This finding emphasizes the importance of fair and transparent cohort selection practices to promote equity in ML-based health care research. Importantly, these results suggest that fairness is not solely a property of the algorithm but is strongly influenced by upstream data construction choices.

Although the AUC is widely used to evaluate model discrimination, it does not capture several critical dimensions of model performance, including calibration, subgroup-specific error patterns, and fairness across populations. As a result, models with similar AUC values may differ substantially in terms of reliability and equity. To address this limitation, this study incorporates additional evaluation dimensions, including fairness metrics and calibration analyses, to provide a more comprehensive assessment of model performance.

Furthermore, although confidence intervals and formal statistical tests are important for quantifying uncertainty, the large sample size of the N3C dataset inherently provides high statistical power to detect differences in AUC, as demonstrated in prior methodological studies. However, due to the computational complexity associated with large-scale data, resampling-based approaches such as bootstrapping were not feasible within the study environment [49]. Future work should explore scalable and computationally efficient methods for uncertainty estimation in large clinical datasets. So, these findings underscore that fairness assessment should not rely solely on discrimination metrics such as AUC, but instead should integrate multiple complementary measures to ensure a robust, reliable, and equitable evaluation of model performance.

Threshold-dependent metrics, such as false positive and false negative rate parity, provide important insights into subgroup-specific error disparities but require the selection of operating thresholds, which may vary across settings. In this study, we focused on threshold-independent metrics and calibration for consistency across cohorts. Future work will extend this analysis to include threshold-dependent fairness measures.

Importantly, this study was designed to evaluate how alternative cohort construction decisions affect model performance and subgroup estimates, rather than to identify a single optimal cohort definition. Therefore, the findings should be interpreted as demonstrating sensitivity to cohort construction choices, not as establishing a validated guideline for fair or standardized cohort selection. Development of such guidelines would require additional clinical validation, external replication, and evaluation across multiple datasets, outcomes, and health care settings.

The study also highlights the challenges posed by inconsistencies in cohort construction practices and the absence of standardized guidelines, which can hinder the comparability and validation of ML models across different settings. Addressing these challenges requires a structured and transparent approach to cohort definition, accounting for equity, clinical relevance, and methodological rigor. From a broader perspective, these findings suggest the need for future work to develop and validate standardized cohort design frameworks that may improve consistency across studies and support more reliable cross-study comparisons.

This research underscores the importance of considering cohort definition and data processing decisions when designing clinical ML models, especially in the context of equity and performance consistency. In addition to these contributions, several limitations should be considered when interpreting the findings. The study shows that cohort selection criteria can influence ML model performance and demographic subgroup results in clinical outcome prediction. Decision D (time inclusion/exclusion) was consistently associated with variation in model performance, particularly in Set 1, while other decisions, such as provider filtering, also contributed to performance differences in Set 2. The findings suggest that certain data processing decisions may affect demographic groups differently, highlighting the need for transparent reporting and sensitivity analysis in cohort construction. By systematically analyzing combinations of data processing criteria across multiple modeling approaches, this work provides insight into the potential consequences of cohort definition choices. This study relied on traditional ML methods; future work should examine whether similar patterns are observed with deep learning–based approaches.

Feature importance derived from tree-based models may be sensitive to data structure and model initialization and may exhibit variability under different sampling or training conditions. In addition, such measures can be influenced by correlated or structured features, which may introduce bias in the relative importance rankings. Therefore, the reported importance values should be interpreted with caution and should not be considered as evidence of causal relationships. Although consistent patterns were observed across multiple models and analytical approaches in this study, future work should further assess the stability and robustness of these findings using repeated resampling strategies, multiple random seeds, and complementary importance measures such as permutation importance or Shapley Additive Explanations (SHAP) values.

Some demographic subgroups, particularly Native Hawaiian and multirace, had very small or zero sample sizes in certain cohorts. In several cases, these groups were absent from either the training or testing sets, making it impossible to compute reliable performance metrics such as AUC. Consequently, reported values of zero or undefined AUC reflect data limitations rather than true model performance. Small sample sizes increase variability and reduce the statistical reliability of subgroup-level estimates, which may lead to unstable or misleading comparisons. Therefore, results for underrepresented subgroups should be interpreted with caution, and fairness assessments are more reliable for groups with sufficient representation. This limitation highlights the importance of adequate subgroup sample sizes when evaluating model performance across demographic populations.

Subgroup fairness analyses were affected by small sample sizes and limited outcome variation in some demographic groups. Although Multimedia Appendix 1 reports cohort-level sample sizes, train/test splits, and demographic distributions across the 16- and 64-cohort configurations, some subgroups, particularly multirace and Native Hawaiian patients, had very small counts in several cohorts. Because subgroup AUC requires both positive and negative outcome cases, and threshold-based fairness metrics depend on the distribution of observed outcomes and predicted classifications within each subgroup, estimates for sparsely represented groups may be unstable. Therefore, subgroup fairness results for small demographic groups should be interpreted cautiously as descriptive indicators rather than definitive evidence of model fairness or unfairness. Future studies should build on these descriptive summaries by reporting subgroup-specific outcome counts and uncertainty measures alongside fairness metrics.

This study has several limitations that should be acknowledged. First, use of data from a specific time period (August 1, 2020-December 31, 2021) may not capture the changing nature of COVID-19, including changes in virus variant, vaccination rates, and health care practices. Therefore, our findings may not be generalizable to current or future scenarios. Second, the study is limited to adult patients (18 years and older), thereby excluding pediatric populations. Accordingly, the results cannot be generalized to younger populations, and differences in the presentation and outcome of COVID-19 among children and adolescents were not examined.

Although 18 years is widely used as the standard threshold for defining adult populations in clinical research and public health reporting, alternative age thresholds could result in modest differences in cohort composition. Future studies could evaluate the sensitivity of the findings to alternative age cutoffs to further assess the robustness of the observed results.

Finally, while the study simplifies data processing decisions to maintain analytic feasibility, limiting the study to only including other possibly influential factors such as socioeconomic status, comorbidities, or geographic variations would give a more complete picture of model performance and demographic disparities.

In conclusion, this study demonstrates that cohort selection is not merely a preprocessing step but a fundamental driver of ML performance and fairness in clinical research. Overall, the study calls for the development of comprehensive guidelines for cohort construction and the validation of existing models against diverse benchmarks to ensure ML models are valid, equitable, and reproducible. These efforts are essential for achieving trustworthy and impactful ML applications in health care, particularly in the context of the COVID-19 pandemic and beyond. More broadly, these findings highlight that improving equity and reliability in clinical ML requires careful attention to upstream data decisions, emphasizing transparency, standardization, and reproducibility as core principles for future research.