Delirium is an acute, temporary, and fluctuating disturbance of attention and cognition triggered by an underlying medical, neurological, or pharmacological condition [1]. With a frequency ranging from 29% to 64% in general medical wards, up to 50% following high-risk surgical procedures, and up to 75% in patients receiving mechanical ventilation in the intensive care unit, delirium is highly prevalent among hospitalized older adults [1-4].

Delirium onset follows dynamic vulnerability-stress models that integrate complex predisposing and precipitating factors. Baseline vulnerability includes preexisting cognitive impairments, affective comorbidities (advanced age), functional impairments (physical, vision, and hearing), and underlying biological frailty (such as sarcopenia) [1,5,6]. These baseline vulnerabilities interact with acute physiological stressors and organic causes: major surgery, polytrauma, infection/sepsis, dehydration, drops in hemoglobin level, dynamic fluctuations in anticholinergic burden resulting from inpatient pharmacotherapy [1,7], as well as prolonged fasting periods (potentially lasting several hours or even days, eg, due to delayed anesthesiology clearance in older patients or limited operating room capacity). Disorientation related to hospitalization itself plays an important role, especially if essential personal aids, such as hearing devices or glasses, are forgotten at the patient’s home or in their usual care environment at the time of acute admission [1].

Delirium itself is associated with an increased risk of mortality and long-term adverse outcomes, including a higher likelihood of developing dementia and requiring institutional care [8].

It is one of the major complications after surgery in older adults who are hospitalized. Furthermore, pharmacological treatments—such as antipsychotic medications—have often proven to be insufficient and ineffective in the management of delirium [9]. In contrast, multicomponent nonpharmacologic delirium prevention interventions have proven effective, reducing the incidence of delirium itself [10]. Furthermore, this multimodal nonpharmacologic approach has been associated with a trend toward a 0.33-day shorter length of stay and a 6% lower likelihood of institutionalization [10]. This is particularly relevant, given that 30% to 40% of delirium cases can be addressed before the syndrome’s onset [3,11].

There is no gold standard tool available for delirium prediction with sufficient diagnostic accuracy [12]. The DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]) is commonly used as a reference standard [13]. DSM-5 is seldom used in routine hospital practice due to its complexity, time, and the necessity of trained clinicians or certified psychologists [14]. Consequently, several delirium assessment tools with higher practical applicability and smaller range of items and points to assess were developed. These tools include the confusion assessment method [15], the Nursing Delirium Screening Scale [16], and the 4AT score [17].

Because preventing the onset of delirium is the primary goal, detecting at-risk patients is of utmost importance [3,11]. When direct prevention as exemplified by the Elder Life Program is not possible, early prediction is crucial [11]. The Elder Life Program includes an orientation protocol, vision protocol, hearing protocol, therapeutic activities, early mobilization, volume repletion, and a nonpharmacologic sleep protocol [11,18]. However, the effective implementation of delirium prediction, treatment, and prevention strategies is still a major challenge in health care all around the globe [9]. In general, strategies that solely rely on delirium prediction and subsequent management may be less effective than risk prediction approaches that allow for the identification of patients at high risk of developing delirium [19,20]. Numerous studies have shown that machine learning models generally outperform rule-based approaches; however, clinical validation within the context of delirium risk prediction is still needed [20-22].

A scoping review identified the following key challenges [23]. First, the underlying dataset frequently lacks unstructured clinical data, which would theoretically require context-sensitive transformer models with strong protections against label leakage for adequate processing [24]. Furthermore, many models rely exclusively on static electronic health record (EHR) snapshots, without accounting for the temporal dynamics of patient physiology [24]. Another recurring issue across included studies is the predominant use of critically ill patient cohorts, limiting broader clinical applicability across more heterogeneous populations. Furthermore, while many of the tools reviewed demonstrated promising predictive performance, the majority have not been prospectively validated or adapted under real-world clinical conditions. Notably, predictive performance frequently degrades when models are applied to patient populations that differ from the original training cohort with respect to age, sex, and comorbidity distribution, underscoring the critical importance of site-specific calibration prior to clinical deployment [24].

Common challenges associated with AI-based tools in medicine frequently concern the collection of data required for model development, as well as the limited availability or restricted access to such data. Furthermore, concerns regarding data security and patient privacy are frequently raised. The volume of data, as well as specific inclusion and exclusion criteria applied to the datasets used for training AI models, may introduce bias. One of the most prominent criticisms, addressed in a 2021 review, is the “black box” problem inherent to many AI systems: such tools are often unable to provide transparent reasoning for the predictions they generate [25].

The aim of this paper is to compare a well-established delirium screening approach (4AT score) with an AI-supported prognostic tool designed to predict patients at high risk at an early stage. Second, the goal of this study is to demonstrate that shifting from a purely questionnaire-based screening approach toward AI-supported delirium risk prediction could enable efficient achievements, including optimized clinical workflows, reduced misclassification, and earlier and more targeted interventions, possibly leading to improved long-term patient outcomes.

The current study is part of a multiprofessional project established to validate the AI-based delirium prediction tool and to implement multicomponent interventions for the prevention of delirium in an Austrian university hospital.

All acute and elective patients admitted to a university hospital trauma ward were included in the study between June 1, 2024, and May 31, 2025.

No formal sample size was calculated, and all eligible patients admitted during the prespecified 12-month period were included to maximize the number of observed delirium events.

For improved readability, delirium onset/diagnosis during the inpatient stay will hereafter be referred to as acute delirium.

For delirium labeling, the EHR was searched for delirium-related codes, specifically F05 (delirium, not induced by alcohol or other psychoactive substances) and F10.4 (mental and behavioral disorders due to use of alcohol: withdrawal state with delirium). In addition to ICD (International Classification of Diseases) code–based identification, the labeling process also incorporated physician discharge letters documenting a clinical delirium diagnosis in narrative form (ie, noncoded diagnoses without a corresponding ICD code), nursing diagnoses of acute confusion (similarly without a formal ICD-coded diagnosis), psychiatric consultation notes, and cases in which patients had received a coded F05 diagnosis during a prior hospitalization. Regarding the latter criterion, the decision was made on the basis that, as the primary focus of this study is risk prediction, patients with a history of prior delirium are at substantially elevated risk of experiencing a subsequent episode and are therefore classified as high-risk patients by default [26]. For the 21 patients classified as positive for delirium based solely on a documented diagnosis of delirium during a previous hospital admission, this approach may introduce a degree of circularity, as delirium history was also incorporated as a predictor in the AI model. Therefore, a sensitivity analysis excluding these 21 cases was additionally performed to test the robustness of the model’s performance.

The method of outcome ascertainment (ICD coding, discharge letters, psychiatric consultations, and nursing diagnoses) was applied uniformly across the entire patient population, irrespective of age, sex, or other sociodemographic characteristics.

The 4AT is a specific screening tool for delirium and is routinely used [17]. The resulting score consists of 4 components. Component 1 evaluates the patients’ level of alertness. Components 2 and 3 are brief cognitive screening tests: the Abbreviated Mental Test 4 and the Months Backwards test (attention assessment). Component 4 assesses acute changes or fluctuations in mental status (Table 1). The total 4AT score ranges from 0 to 12. A score of 0 indicates that delirium and/or moderate-to-severe cognitive impairment is unlikely. Scores of 1 to 3 may reflect significant cognitive impairment. A score of 4 or higher indicates possible delirium [17]. During the study, assessments were carried out on multiple occasions: at the time of admission, on the evening of the day of admission, and on the second day of admission. Furthermore, repeated 4AT assessments were conducted in patients with elevated scores (≥4) or in cases where clinical deterioration was observed. However, for the purpose of this study and to ensure a comparable statistical approach with the prediction tool, only the highest score obtained on the day of admission was used for analysis. All 4AT assessments were performed by ward nursing staff.

EHR data required for the analysis were extracted from the HIS. This included information used for delirium labeling (corresponding ICD-10 codes, discharge letters, psychiatric consultations, and nursing diagnoses), demographic data such as age and sex, as well as transfer data for all patients included in the analysis. Individual risk predictions generated by the Personalized Risk Tool were extracted from the on-premises database and linked to the study cohort. All 4AT scores were available from the HIS. Patients with a missing 4AT score or a missing AI-based risk prediction were excluded from the analysis.

The study was approved by the local institutional ethical review board (reference number 30‐146 EX 17/18).

We aimed to test and evaluate an AI-based risk prediction tool in routine clinical practice and to compare its performance against a well-established diagnostic screening instrument (4AT score).

The risk prediction demonstrated a sensitivity of 87.3%, while the 4AT score showed a sensitivity of 28.0%. Regarding patients without delirium, the AI tool’s prediction showed a specificity of 73.9%, compared to the specificity of 96.3% achieved by the 4AT score. The discriminative performance of the AI-based tool achieved an AUROC of 0.884, indicating excellent discrimination, with an 88.4% probability of correctly categorizing patients developing delirium as at-risk. The narrow CI detected showed high statistical precision. Additionally, given that the entire interval substantially exceeds the 0.8 threshold, the result can be considered robust and clinically meaningful. In comparison, the 4AT score shows a notably lower AUROC, which reflects acceptable but moderate performance as an early detection tool. As the 2 CIs do not overlap, the first model demonstrates clear superiority.

This study has several limitations inherent to the analyzed topic and study design. The most important limitation is the fundamental difference in purpose between the 2 instruments evaluated. The 4AT score is a validated diagnostic screening tool intended to detect current delirium or acute cognitive fluctuations, whereas the AI-based model is a prognostic tool designed to estimate the risk of developing delirium during hospitalization. In the present study, we compared the 4AT score’s performance with an AI-based prognostic tool: a reference standard was required against which the performance of the AI-based prediction model could be benchmarked. In the absence of a more suitable comparator, the well-established 4AT score served this purpose. Moreover, the 4AT questionnaire is, by definition, a screening instrument designed to assess the patient’s current delirium status, but it also allows to observe fluctuations and changes in the patient’s neurological status over time and should be interpreted as such. In the context of this study, the 4AT score was evaluated under conditions that may not fully reflect its intended use in clinical practice, as only the highest score on the day of admission was considered.

Consequently, the 2 approaches address different clinical questions and should be regarded as complementary rather than directly competitive. This study’s results suggest that the AI-based approach may provide greater predictive value at the time of admission for identifying patients at risk of developing delirium during hospitalization, while the 4AT remains an important tool for ongoing clinical screening during the hospital stay.

Furthermore, the performance of the AI model is dependent on the quality, completeness, and consistency of the underlying clinical data. Variability in documentation and coding quality represents a key success or failure factor for the AI-based tool, as its predictions are inherently dependent on the accuracy and consistency of the underlying clinical data in the HIS. This could have led to documentation, selection, and cotreatment biases in the present study.

A total of 638 (31.8%) patients were classified as having a high or very high risk by the AI-based tool. Of these, 473 patients did not develop delirium during the hospital stay. This corresponds to a false-positive rate of 26.1%. However, this does not imply that these patients were not at risk of delirium. In some cases, precipitating factors may have been absent during the current hospital stay, and, consequently, the patients did not develop delirium. Moreover, the ultimate goal of such a prognostic AI-based tool is to trigger targeted multidisciplinary preventive measures and, ultimately, to minimize the onset of delirium. Therefore, if an increasing number of at-risk patients do not develop delirium and the false-positive rate increases, the aim of such an AI-based prognostic model is achieved. However, the impact of such risk stratification through possible targeted preventive interventions was not evaluated in the present study.

Although the absence of a high or very high delirium risk according to AI risk stratification does not imply the absence of delirium risk, alert fatigue is a crucial aspect to consider and may evolve over time. Resources in hospital wards are limited, and it may not be feasible to implement preventive measures for 31.8% (n=638) of patients in a ward. In such cases, the AI-based tool could be recalibrated, and the number of patients assigned to each risk category could be adapted. This could reduce the number of risk alerts.

The cohort analyzed differed from the development cohort with respect to center composition. The evaluation cohort was derived from a single center, whereas the development cohort was based on multiple centers. The development dataset relied on ICD-10 coding, whereas the evaluation dataset additionally incorporated noncoded clinical documentation (discharge letters, nursing diagnoses, and psychiatric consultations) to identify cases of delirium.

A systematic review and meta-analysis from the year 2022 for the 4AT score show a pooled sensitivity and specificity of 87% and 87%, respectively. In combination with its conciseness and simplicity, it confirms itself as an established and useful tool [23].

As for the AI, a systematic review and meta-analysis of the year 2022 for machine learning–based prediction models for delirium show that the overall pooled AUROC for predicting delirium was 89%, with sensitivity of 85% and specificity of 80% [31].

Compared to the pooled values reported, the AI tool used in this study demonstrated a higher sensitivity, exceeding the pooled values reported in the recent meta-analysis. This, however, comes at the trade-off of lower specificity.

Notably, the 4AT score performed considerably below the values reported in prior studies [17,23], which is attributable to the fact that only the single highest value recorded on the day of admission was used in this analysis (as a benchmark for the AI model). Of note, the 4AT score is, by definition, a screening instrument designed to assess the patient’s current delirium status and is used multiple times a day to capture fluctuations. Therefore, these results do not call into question the quality of the 4AT score.

It should be noted that the 4AT score is a questionnaire-based tool, which considerably increases the costs and infrastructure associated with its administration. From a practical standpoint, the AI-based tool offers easier implementation in the overall workflow than the 4AT score. While the 4AT score requires active administration by nursing staff, followed by manual documentation and transfer into the EHR, the AI-based tool operates automatically in the background, functioning both as a decision-support tool and a source of clinical information. On the other hand, the AI-based tool grounds its predictions and prognoses on the information documented (correctly) in the HIS. Therefore, a drawback for the AI-based tool is the absence of standardized diagnostic criteria for delirium, resulting in ICD-coding being applied less consistently in patients with confirmed delirium compared to other medical conditions. One contributing factor is that delirium diagnoses often rely on subjective clinical assessment of the patient’s condition, a challenge that is further compounded by the varying subtypes (hypoactive, hyperactive, and mixed forms) and severity levels of delirium [20,26].

In general, other studies addressing AI-based delirium diagnostics using EHR data continue to face certain limitations, several of which are at least partially addressed by our tool.

A notable strength of the present study (in contrast to the majority of comparable prior studies) is that the AI prediction tool was validated and tested under real-world clinical conditions.

Nevertheless, further studies that include varying cultural backgrounds and differing health care systems remain warranted. AI models have the potential to support health care providers in their daily clinical tasks. By assuming a variety of responsibilities, AI-based tools may contribute to a reduction in overall workload, enabling providers to concentrate their efforts on a more focused set of tasks [25].

The AI-based delirium prognostic tool demonstrated superior early predictive capacity for delirium risk at admission, compared to the 4AT score. Importantly, these 2 approaches address distinct clinical questions and should therefore be considered complementary rather than directly competing. This distinction is particularly relevant because no sufficiently comparable, well-established prediction tool currently exists against which this AI-based tool can be directly evaluated. A notable strength of the present study was that the AI prediction tool was validated and tested under real-world clinical conditions. The AI-based tool showed high sensitivity, exceeding values reported in prior meta-analyses, but with a trade-off in specificity [31]. The resulting insights may enable proactive risk management and targeted risk reduction.