Artificial intelligence (AI) is expected to benefit the medical domain with improvements in patient care and reduced workload for medical personnel. AI models are increasingly proposed as a knowledge or inference base of clinical decision support systems (CDSSs) to achieve this. These AI-based CDSS are considered well-suited for use in the intensive care unit (ICU) due to the amount of available documented data in the ICU, but the transfer of these systems into clinical practice is still lacking [1]. AI models for which it is impossible to directly infer why they generate an output due to their complexity are called black-box models. The black-box nature of these models poses a barrier to adoption in the medical domain, as it makes it difficult for health care providers to understand, trust, or accept these models [2-4]. According to the EU (European Union) AI Act, AI-based CDSS are classified as high-risk AI systems [5]. For high-risk AI systems, the act requires the possibility of human oversight and the provision of appropriate interfaces for this supervision (EU AI Act, Chapter III, Article 13 and 14) [6]. Approaches from the field of explainable AI (XAI) are a solution to these barriers [7-11]. XAI provides additional information to the output of an AI model to make model reasoning more transparent through explanation user interfaces (XUIs). In the medical domain, XAI has been shown to influence users’ trust [12,13], the perceived usefulness [14] or value [13] of a system, the medical professionals’ confidence in decisions [14], and decision-making [15]. XAI research focuses on creating, optimizing, and evaluating XAI algorithms. Explanations are often designed to meet the needs of (X)AI developers while neglecting the needs of clinical end users [16]. Recently, calls for a user-centered design (UCD) of XUIs have been raised in the medical informatics domain [16-20].

The objective of this paper was to conduct the initial design and evaluation phases of the UCD process for an XUI of an AI-based CDSS intended for use in the ICU. As a use case, we selected the provision of recommendations for ventilation parameter settings in patients receiving mechanical ventilation in the ICU. Patients may undergo mechanical ventilation using ventilators—machines that assist breathing by delivering air into the lungs when they cannot breathe adequately. Another form of life support is extracorporeal membrane oxygenation, which bypasses the lungs—and sometimes the heart—by circulating the patient’s blood through an external machine that oxygenates the blood and extracts carbon dioxide from it. To enable the continuous optimization of a patient’s ventilation, a CDSS must be able to recommend a range of parameters. In mechanical ventilation, these include, for example, the ventilator mode, which defines the overall strategy and level of mechanical support; tidal volume, which specifies the amount of air delivered per breath; and respiratory rate, which determines the number of breaths per minute.

The development of such AI-based CDSS is being carried out within the IntelliLung project, a European Union–funded initiative aimed at improving ventilatory care and reducing the workload of clinical staff [21]. The CDSS will generate recommendations for both mechanical ventilation and extracorporeal membrane oxygenation settings. The research presented in this paper used the IntelliLung CDSS as a real-world use case to anchor the design in clinical practice. However, the design is independent of any project-specific AI algorithm decisions and instead focuses conceptually on XUI design for such a CDSS. As Mohseni et al  [22] proposed, this type of research can be conducted before selecting and developing the AI algorithms and corresponding XAI methods.

Although much research on XAI focuses on the health care domain [23], to the authors’ knowledge, no paper explicitly addresses the UCD of XUIs for AI-based CDSS aimed at optimizing continuous mechanical ventilation in the ICU. Several papers report on the use of AI algorithms and XAI methods from a technical perspective, for example, to predict the ventilation duration for patients with acute respiratory distress syndrome [24], to predict the disease progression of patients with acute respiratory distress syndrome [25], to predict the ventilator support for patients with COVID-19 [26], and to generate ventilator setting recommendations [27]. Hughes et al [28] report on the context and work process analysis for CDSS supporting the weaning process in the ICU to inform the overall CDSS design for weaning support, but do not focus on optimizing continuous ventilation or designing XUIs for the CDSS.

A limited number of papers address the UCD of XUIs in the ICU [14,29-31]. A preprint by Clark et al [29], published after this study concluded, investigates user requirements for using XAI in CDSS, without focusing on a specific ICU use case. Nagendran et al [30] demonstrated the feasibility of eye-tracking analysis to assess physicians’ interactions with an XUI for an AI-based CDSS, although not focused on ventilation, in a simulated ICU setting. Barda et al [31] designed and evaluated an XUI for an AI-based CDSS that predicts mortality risk in a pediatric ICU. The acceptance of an XUI for a sepsis treatment CDSS by ICU clinicians was investigated by Sivaraman et al [14]. Outside the ICU setting, Schoonderwoerd et al [32] investigated pediatricians’ perspectives on different explanation concepts for a diagnostic CDSS.

The UCD process was conducted in accordance with DIN (Deutsches Institut für Normung) EN (Europäische Norm) ISO (International Organization for Standardization) 9241-210 [33] to design the XUI for an AI-based CDSS that recommends ventilator settings. This paper focuses exclusively on the first iteration of the design and evaluation phases of the UCD process. These phases rely on the results of the context analysis and the specification of user requirements. The results of the first 2 phases of the UCD process are summarized to provide the necessary context.

During the context analysis, the intended use characteristics of the CDSS were defined as (1) a stand-alone CDSS, which should not replace the ventilation monitor, and (2) a CDSS that would be displayed on a stationary tablet next to the ventilation monitor of an individual patient. The identified main user groups of this CDSS were resident physicians and ICU nurses with the authority to change the patient’s ventilator settings. An initial investigation into user requirements revealed conflicting opinions among future users about the importance of explanations [34]. These results led to a follow-up investigation into users’ preferences for explanation, to identify which explanations to include in a first design iteration. Seventeen explanation concepts, ranging from less complex ones such as available input information to more sophisticated ones such as counterfactual explanations, were presented to the users as low-fidelity mockups in a questionnaire. This step was performed before this study to attain a manageable number of explanation concepts for the first design iteration and a formative evaluation. The selection of explanation concepts for this first iteration of the XUI was based solely on user preferences, rather than on commonly used explanation concepts. Users selected the explanation concepts, output certainty, and outlier warning to be displayed alongside the CDSS recommendations. The output certainty concept informs users about the CDSS’s confidence level in its recommendation. The outlier warning alerts users if one or more input parameter values are highly unusual. Three additional explanation concepts were selected but were not designated to appear directly next to the CDSS output: available input, feature importance, and rule-based explanation. The available input explanation concept informs the user which parameter values were available when generating the recommendation. The feature importance explanation concept shows users how each parameter influenced the recommendation. The rule-based explanation concept presents simplified rules in text or decision tree format, approximating how the recommendation was calculated. These findings served as the foundation for the XUI design phase.

The design phase began with the development of a foundational knowledge base for the XUI. Relevant literature, such as norms and design guidelines, was reviewed. For this purpose, the ISO 9241 standard series on human-system interaction and human-centered design was reviewed. This foundation was expanded through a scoping review and the development of design recommendations for the user-centered XUI design for CDSS [35]. Searches for research papers related to the topic of XUIs for AI-based CDSS for the ventilation of patients in the ICU found no relevant results before the design of the XUI.

Midfidelity XUI mockups were developed using the Justinmind [36] prototyping software, based on the knowledge foundation. The design of the mockups was intended to direct users’ attention to elements relevant to RQ2 and RQ3 and to prevent users from focusing on aesthetic aspects such as the color of the XUI. Therefore, mockups were designed with the following characteristics: (1) realistic variable names and content to support user familiarity; (2) realistic-looking data, for which synthetic data points were generated for each displayed input variable with a large language model (Claude Sonnet 3.5 [37]); (3) minimal use of color, used selectively to highlight system functionality; and (4) simple interactions (eg, navigation between interface levels).

A main screen for the CDSS was designed to present the XUI within its intended system context and to help users envision the future context of use for the XUI. The basic structure of the XUI and the design variants for the explanation concepts were developed based on the list of selected explanation concepts and their intended level within the XUI, which had been defined during the requirements specification phase of the UCD process. The mockups were designed iteratively: the first author created initial drafts, and members of the author team with usability expertise provided verbal feedback for improvements during multiple group meetings.

The design of the individual explanation concepts was oriented toward typical design elements used for these explanation concepts in the XAI literature. This decision was made to provide the users with a realistic representation of common explanation concepts and to keep the focus on the goal of this first evaluation iteration. We opted not to apply a more elaborate design strategy to the explanation concepts in this iteration; this will be addressed in later iterations, which typically focus on aesthetics and detailed design.

In the evaluation phase, formative feedback was gathered from ICU physicians and nurses, focusing on the XUI’s structure and explanation concepts. A formative evaluation was chosen to collect insights for improving the CDSS and understanding users’ initial impressions of the CDSS. Given resource constraints—such as limited availability of clinical experts and their high workload—a pragmatic approach was taken. This evaluation is reported according to the STARE-HI (Statement on the Reporting of Evaluation studies in Health Informatics) guidelines to ensure consistency and standardization [38]). For the STARE-HI reporting checklist, refer to Multimedia Appendix 1 and [39].

As a formative evaluation, 2 group usability walkthroughs were conducted—1 for each main user group (physicians and ICU nurses). A semistructured approach was followed, guided by a predefined protocol. Each walkthrough included (1) a presentation of the XUI, (2) individual reflection time, (3) guided group discussions, and (4) a quantitative assessment (QA) of the explanation concepts.

The walkthrough groups were separated by user group. This separation accounted for differing levels of project involvement and aimed to minimize hierarchical bias, particularly the risk of physicians influencing ICU nurses during discussions. The walkthroughs were moderated by a researcher with usability expertise, who introduced the mockup, guided the discussion using predefined questions focused on information retrieval, overall interaction, and the structure of the XUI, and asked follow-up questions where necessary. A second researcher took notes to ensure thorough documentation.

The target was to recruit 4 to 5 participants per user group, as small sample sizes are considered acceptable for formative evaluations in medical informatics [40], and sample size requirements can be adjusted based on system-specific characteristics [41].

Participants were recruited using convenience sampling. For the physician group, invitation emails were sent to 5 physicians in the same research group, which was involved in the IntelliLung project. Four physicians accepted the invitation. To increase the chances of successful recruitment of ICU nurses, 1 of the cooperating physicians identified an interested contact person among the nursing staff in 1 of the ICUs at the same hospital. The contact person informally inquired about his or her peers’ interest in participating in the study. After the nurses were recruited, the contact person shared the contact details with the researchers. To be eligible to participate in the study, all participants were required to have at least 3 months of ventilation experience (demonstrated through working experience in the ICU or research activity in the field of mechanical ventilation). No compensation was paid for participation in the group usability walkthrough.

The details of the physician walkthrough are described in this section. Any predefined process deviations for the nurse walkthrough are reported in the section “Deviations for the Nurse Group Usability Walkthrough.”

At the start, participants received in-depth information about their rights and the purpose of the usability walkthrough. Afterward, the moderator presented the CDSS use case and the main screen of the CDSS. The XUI mockup evaluation then began. The 2 XUI levels were evaluated sequentially, reflecting their hierarchical structure.

The evaluation began with the main CDSS screen, which included XUI level 1, displayed on a monitor. The moderator provided a verbal introduction. Participants were then given time to observe and reflect on XUI level 1 silently. The reflection was followed by a group discussion guided by the question: “Is the presentation of outlier warning and system certainty on the main screen sufficient?”

The XUI level 2, a subordinate screen providing additional explanation concepts, was evaluated in 2 steps. In the first step, the participants assessed the overall structure of the level. The XUI level 2 was shown, and the participants silently analyzed it. Afterward, the group briefly discussed and then collectively answered the following questions: “In which part of the interface do you find explanations for the recommendation?” and “How do you switch between the explanations?”

In a second step, the individual explanation concepts in XUI level 2 were assessed using the following steps:

The complete version of the instrument is provided in Multimedia Appendix 2.

The group usability walkthrough concluded with a general discussion focusing on the following guiding questions: “Should there be more or fewer explanation concepts presented?” and “Are there any missing explanation concepts?” (The physicians had access to previous presentations of other explanation concepts before the group usability walkthrough).

To accommodate the ICU nurses’ limited previous exposure to the project, the walkthrough session was adapted to include a more comprehensive introduction to CDSS and AI concepts. Insights from the physician group informed minor refinements to the mockups. Elements that had proven consistently irrelevant or confusing across previous sessions were excluded to maintain focus, while still allowing nurses to contribute unique, user-specific feedback.

Furthermore, in the final group discussion, the nurses were verbally introduced to the possibility of additional explanation concepts (since they had not previously been exposed to other types, such as counterfactual explanations). They were also asked whether the information presented in level 1 of the XUI was sufficient.

This study was conducted in accordance with ethical research principles. It involved low-risk, formative group usability walkthroughs as a qualitative survey or interview method in which clinician participants were shown interface mockups and asked structured questions about usability, understandability, and suitability. No medical interventions were performed, no patient data were collected, and the study did not constitute epidemiological research. In line with the institutional guidance of the Ethics Committee of TU Dresden for such non-interventional survey or interview studies, no IRB or REB approval was sought [42]. All procedures were carried out in accordance with the principles of the Declaration of Helsinki. Informed consent was obtained from all participants after they received detailed information about the study purpose, the walkthrough procedure, data handling, and their rights. Participation was voluntary, and participants could withdraw at any time without consequences. Participants were not compensated for their participation. Privacy and confidentiality were safeguarded by collecting and storing data in pseudonymized form; only non-sensitive, profession-related participant information was collected, no identifying information is reported, and access to raw data was limited to the research team.

The design phase produced a 2-level XUI structure (XUI level 1 and XUI level 2). XUI level 1 is positioned on the main screen of the CDSS, adjacent to the system’s recommendations. Figure 2 shows the mockup for the CDSS main screen, which contains a header with basic patient information (A in Figure 2), the recommendation (B in Figure 2), XUI level 1 (C in Figure 2), and a trend view of important parameters (D in Figure 2). XUI level 1 includes the explanation concept outlier warning (E in Figure 2), consisting of an icon with a header (H in Figure 2) and an information text (I in Figure 2); the explanation concept certainty information (F in Figure 2), for which the certainty level is presented as a traffic light (J in Figure 2), a number, and a category label (K in Figure 2); and a button (G in Figure 2) that links to XUI level 2.

Figure 3 shows XUI level 2, which includes a header with basic patient information, the CDSS recommendation, a repetition of the explanation concepts from level 1, and an expanded panel on the right side for displaying explanation concepts in more detail, one explanation concept at a time. The following explanation concepts were used in level 2: available input, feature importance, and rule-based explanation. The users can switch between explanation concepts in the expanded section using tabs.

Figure 7 provides the QA results (QA-1 to QA-3) for the available input and feature importance concepts.

As shown in the top half of Figure 7, the physicians agreed or strongly agreed that the available input concept was understandable, suitable for clinical work, and visually appealing. The nurses mostly disagreed or responded neutrally regarding the concept’s suitability and understandability, and found the visualization aesthetically unappealing. The nurses’ and physicians’ ratings, therefore, differ across all assessed dimensions, with nurses being more critical overall.

The QA results for the feature importance explanation concept are shown in the bottom half of Figure 7. Most physicians strongly agreed that the feature importance concept was appealing, suitable, and easy to understand. The nurses mostly agreed with its understandability and appeal, but tended to disagree with its suitability for clinical work. Both user groups, therefore, rate the understandability and visual appeal of the explanation concepts positively. However, the user groups differ in their assessment of the suitability for their clinical work, with a positive assessment by the physicians and a negative assessment by the nurses.

Figure 8 provides the QA results for the rule-based explanation concept. Three physicians strongly agreed or agreed with its appeal and understandability; one chose the “no answer” option. All 4 physicians disagreed or strongly disagreed with its suitability for clinical practice.

The calculated descriptive statistics for each QA for all 3 explanation concepts on the second level are provided in Multimedia Appendix 3. The descriptive statistics mirror the results visible in Figures 7 and 8.

This paper provides findings from the first design and formative evaluation phase of an XUI for an AI-based CDSS aimed at optimizing mechanical ventilation in the ICU.

The design led to an XUI with 2 levels. Level 1 of the XUI, positioned next to the CDSS recommendations on the main screen, includes the output certainty and outlier warning concepts. Level 2 of the XUI provides on-demand access to additional explanation concepts: available input, feature importance, and rule-based explanation. Multiple versions of the explanation concepts on the second level have been designed and evaluated.

The initial evaluation consisted of 2 group usability walkthroughs with ICU physicians and nurses. Both user groups liked the first level of the XUI. The physicians were satisfied with XUI level 2 but provided valuable feedback for improving it. The nurses were skeptical about whether they would use level 2 of the XUI in clinical practice.

For the design phase, the following research question was investigated: RQ1: “How should the XUI of an AI-based CDSS for mechanical ventilation optimization in the ICU be designed?”

To address this research question, the first focus was on the XUI structure. The 2-level structure of the XUI allowed the provision of easy-to-grasp explanation concepts next to the CDSS recommendation and more complex and more detailed explanation concepts on the second level, on demand. This structure was based on the input of the previous UCD design phases and current design recommendations [35]. This design choice aligns with new recommendations published as a preprint after the work reported in this paper was concluded. The preprint recommends multiple levels with increasing detail for XUIs for the ICU [29].

After the structure of the XUI was created, the investigation addressed the presentation of the explanation concepts themselves. The individual visualizations for the explanation concepts are comparable to typical visualizations found for XUIs in the medical domain [43,44]. The explanation concepts were kept in line with the design of XUIs to provide clinicians with an impression of the possibility of explanations at the current state of the art of research in the field.

The different versions of the explanation concepts in level 2 of the XUI varied in the level of detail presented to determine users’ preferred level of detail for this CDSS. Based on the literature, this is considered an important design decision [29,35]. Minor adaptations were made to incorporate design recommendations for the UCD of XUIs; for example, the explanation concept of feature importance was presented to the user under the term “parameter importance,” as the term feature importance was assumed not to be self-explanatory for non-AI experts. In addition, to add some variety to the different presented explanation concepts, the typical bar plot visualization for the feature importance explanation concept was extended by a version inspired by the works of Bienefeld et al [45], which created a heat map–style feature importance history in an XUI.

During the evaluation phase, the following research question, RQ2, was investigated: “What are clinicians’ perceptions of the proposed XUI of the AI-based CDSS for mechanical ventilation optimization?”

The users’ first impression of the 2-level structure of the XUI suggests that this structure is the right approach for this CDSS. Both groups were content with level 1 of the XUI, and the physician group was satisfied with level 2 of the XUI, while the nurses were more skeptical of their intentions to use level 2. No concerns were raised by either group about the general structure. The structure allows both user groups to access the information they prefer while preventing information clutter on the main screen of the CDSS.

The combined results from group discussions, concept selection, QAs, and free-text responses indicate that ICU nurses and physicians had noticeably different preferences regarding the level of detail and complexity in the XUI. The nurses preferred less complex explanation concepts, such as output certainty and outlier warning, explicitly requested a lower level of detail for the explained concepts, and rated the suitability of the explanation concepts on level 2 lower than the physicians. The literature supports this, as it has been reported that nurses wished for clear and concise explanations, outputs, and actionable information when dealing with AI [31,46]. In contrast, the physicians were satisfied with both XUI levels, which allowed them to consult low-detail and less complex information on level 1 and more detailed and complex explanation concepts on level 2 of the XUI. This implies that the explanation needs and preferences between the nurses and physicians differ, with physicians having a higher need for more complex and detailed explanations than the ICU nurses. The work from Barda et al [31] found a similar insight. Current research supports this by showing that different health care provider groups can have different explanation preferences [15] or that different stakeholders have different explanation needs [47]. The observed differences may reflect role-specific workflows in ventilator care. ICU nurses typically operate in fast-paced bedside routines with frequent monitoring and rapid adjustments, which may favor concise, immediately actionable explanations. Physicians, in contrast, often integrate XAI outputs into broader diagnostic reasoning and decision validation, which may increase the perceived value of more detailed, mechanism-oriented explanations provided on level 2.

Notably, divergence between groups was most consistent for clinical suitability rather than for understandability or visual appeal (Figure 7). This suggests that acceptance may not only be a matter of whether clinicians can comprehend an explanation and perceive it as visually appealing, but also whether it fits their real-world task context and decision-making responsibilities. Therefore, suitability should be treated as a primary design target in XUI development, alongside clarity and aesthetics.

These results emphasize the need for a rigorous application of the UCD process when an XUI is developed for different health care providers. In the XUI developed and evaluated in this study, the 2-level solution appears to meet the differing explanation needs and preferences of the user groups. Future iterations should evaluate whether this approach remains sufficient or if separate XUIs are needed. It should be kept in mind that ICU nurses will probably not consult level 2 of the XUI in future iterations. Therefore, presenting potentially necessary and valuable information only on level 2 should be avoided. Clinical role alone may not fully explain the differences; AI literacy and ventilator-care experience likely act as moderators that shape how much explanation depth is perceived as useful versus burdensome. Future work should disentangle professional role effects from training- and experience-related factors, for example, by stratifying participants accordingly.

Interestingly, both groups were skeptical about their peers’ adaptations of the explanation concepts in the clinical daily routine, which was voiced during the group discussions and in the free-text answers. The nurse group voiced more skepticism than the physician group, and their skepticism increased with the level of detail and complexity of the explanation concepts. The physicians were mainly satisfied with the proposed explanation concepts, apart from the rule-based explanation concept. However, they stressed that the complexity of the explanation concepts should not exceed that used in the XUI. These results explain why both groups had no interest in more complex explanation concepts, such as a counterfactual explanation. This result contrasts with the results of a study by Jung et al [48], in which clinicians had a positive perception of counterfactual explanations.

The physician group positively assessed the explanation concepts on level 2 of the XUI, apart from the suitability of the rule-based explanations. The understandability ratings for the explanation concepts are similar to previous research, showing that physicians can understand the selected concepts [32]. The positive assessment of the feature importance explanation concept for the physician group is in line with other research efforts [14,15]. Due to the overall positive assessment by the physicians of both levels of the XUI and nurses’ positive perception of level 1 of the XUI, it seems that the results of the first design phase were a step in the right direction. However, the evaluation also revealed some improvement opportunities.

Second, the evaluation phase investigated research question RQ-3: “What XUI design improvements should be addressed in subsequent design iterations?”

Based on the results of the evaluation, all the explanation concepts should be revised, keeping the nurses’ preferences for a low level of detail and complexity for the explanation concept, and their perception of the explanation concept as having too much detail in mind. In the following design iteration, all the explanation concept versions should be provided with a lower level of detail or complexity as a default version. From these default versions, the more interested user should be able to access a more detailed version on demand. This approach has to be balanced so that the more interested users are not burdened while accessing the explanation concepts with the current level of detail and complexity for which they provided positive feedback in the evaluation.

For the feature importance explanation concept, the evaluation revealed that further research is needed for the design of the x-axis. The x-axis should be redesigned to be understandable for physicians and nurses and to transport the appropriate amount of information without overwhelming them. As raised in the physician group walkthrough, one option could be an empty x-axis; the suitability of such a version should be tested in comparison to other typical variants of the x-axis for this explanation concept. As the question about the actionability of the information provided in the feature importance explanation concept arose in the physician group, it should be investigated whether incorporating information about how to act on the explanation concept would be valuable. For the outlier warning explanation concept, the evaluation revealed different preferences for the case in which no outlier would be detected. It should be investigated how the user groups would react to the version they did not prefer when interacting with a version of the XUI.

An increased effort should be put into the design of level 1 of the XUI. No alternative versions had been designed for the explanation concepts on level 1 for this initial prototype to reduce complexity. These should be a primary focus for the next design iteration, as both user groups saw them as valuable. While refining these explanation concepts, an emphasis should be placed on concise information visualizations and effortless information retention from both user groups.

This first iteration of the design and evaluation phase of the UCD process for the XUI of a CDSS has some limitations that influence the generalizability of the results and should be carefully considered when interpreting the results of this study. A pragmatic group usability walkthrough was selected for the evaluation phase due to the limited number of available and suitable participants per user group. Nonetheless, this pragmatic approach proved valuable as it allowed for the receipt of early feedback from subject matter experts on mechanical ventilation with real-world and research experience. Their experience might have biased their assessment of the XUI, as they might not have had to rely on the CDSS in the first place due to their extensive experience. However, their knowledge allowed them to focus on the XUI design and incorporate extensive practical knowledge and expertise in their feedback and assessment of the XUI. The small group size of only 4 participants per group usability walkthrough might be seen as a limitation, but this group size allowed for a lively discussion between participants, and smaller sample sizes are in line with early evaluation iterations [40,41]. In addition, the group walkthrough allowed for the collection of feedback in a condensed amount of time, which was necessary due to resource restrictions. Nonetheless, these points limit the generalizability to the clinical population that should be addressed in the following design iterations.

Furthermore, the changes for the nurse walkthrough limit the comparability between the 2 walkthrough groups. The walkthrough had to be adjusted to avoid overwhelming the participants. Despite the minor adjustments, the collected insights allow for a preliminary assessment of the difference between the user groups. No actual data or patient vignettes were used to promote deeper engagement or trigger demand for an explanation from the CDSS, which limits the comparability of the assessment of XUI in real-world situations. The current state of the XUI mockup was sufficient to spark extensive discussions while preventing the participants from focusing on the medical validity of the situation or mockup content. In this initial iteration of the UCD, only subjective assessments of the XUIs were elicited. This may not provide a complete picture, as it has been shown for XUIs in the ICU that subjective assessments might not transfer to the use or influence of the XUI during the medical decision-making process [30,49]. Nevertheless, the amount of valuable insights generated through subjective assessments of the XUI in this study shows that this type of feedback is helpful during such an early design phase. These limitations should be addressed in future evaluation iterations.

Based on this research, the XUI mockup should be improved. The improvements should address the identified opportunities for enhancement in the XUI, making it more suitable for both user groups without drastically altering it, as physicians were already satisfied with the current design. A higher focus should be placed on level 1 of the XUI, as this is likely the most used part. In addition, feedback should be sought from the AI and XAI algorithm developer and incorporated into the mockup to enhance the realism of the XUI for the next iteration. An improved and extended XUI mockup would be a suitable basis for a more elaborate evaluation. The enhanced mockup could be accompanied by different vignettes of patients and situations that might trigger the need for an explanation from the user. Future evaluation rounds should have a higher experimental character to avoid the potential shortcomings of subjective assessments of an XUI in the ICU and allow the users to interact with the XUI directly. The proposed evaluation iterations should aim for a higher sample size and a diversified participant pool concerning their ventilation experience, AI literacy, and clinical center to foster generalizability and reflectiveness of the clinician population. To this end, a remote experimental study could be conducted in which the user directly interacts with the CDSS based on presented patient vignettes. During such an evaluation, participants reflecting a broad base of clinical users should be recruited from multiple institutions.

To the best of our knowledge, this paper is the first to report on the UCD and formative evaluation of an XUI for an AI-based CDSS aimed at optimizing continuous mechanical ventilation in the ICU. The findings indicate that a 2-level XUI structure can effectively address the distinct explanation needs of ICU clinicians. Nurses’ information needs were largely met through concise, high-level explanations such as outlier warning and output certainty, presented alongside CDSS recommendations, whereas physicians preferred access to more detailed explanations—such as available input and feature importance—on a secondary level. These results provide empirical evidence that explanation needs differ by professional role and underscore the necessity of user-centered, iterative design processes in the development of XAI systems for critical care. Future work should refine these explanation concepts and evaluate their impact on clinical decision-making, user trust, and system adoption.