This manuscript follows SUNDAE (Standards for Universal Reporting of Decision Aid Evaluations) guidelines [26]. We conducted a randomized formative evaluation of DDInteract compared to UC from March 2021 to August 2021. Formative evaluation is a research methodology used to rigorously examine factors that might influence the progress during the implementation of health care innovations [27]. We used a mixed methods approach, as we collected both quantitative and qualitative data.

This project was approved by University of Utah Institutional Review Board (00127062). All participants signed the inform consent before participating in this study.

We designed DDInteract to be a free SDM tool that is integrated with the EHR to mitigate risk of harm due to DDIs involving warfarin and NSAIDs [25]. When launched from within an EHR, the tool retrieves risk factors from the patient’s record (eg, history of GI bleeding, age, medications) and calculates the patient’s risk of bleeding. A dynamic risk array displays and updates the risk of GI bleeding in real time based on clinician input, patient history, and medication choices (Figure 1). When an NSAID is selected, the risk array changes to reflect the increase in risk due to the combined effects of both agents on risk of bleeding. The tool also (1) provides information on nonmedication treatments for pain, such as acupuncture and physical therapy; (2) includes strategies to reduce risk of GI bleeding through use of proton pump inhibitors; (3) facilitates the medication prescription process; (4) automates clinical documentation of the SDM discussion; and (5) provides patient education in an “after visit summary.” Further details about DDInteract functionality are described elsewhere, and the tool is available for demonstration purposes on the DDInteract website [28].

We randomized physician-patient dyads to participate in simulated clinical encounters using either DDInteract or conventional drug information resources (UC) to treat a patient on warfarin who is also seeking relief of pain by using an NSAID. Physicians in both groups (DDInteract or UC) received access to various online DDI resources (eg, Micromedex), while those in the intervention group also had access to DDInteract.

We recruited physicians through electronic communications, presentations to medical groups, and snowball sampling. The inclusion criteria for physicians required experience with treating patients on oral anticoagulants. We recruited patients from the University of Utah Health System, including the Division of Cardiology and the Anticoagulation Service. Patient participants needed to be older than 21 years of age, fluent in English, and currently taking an oral anticoagulant.

We integrated and deployed DDInteract with the Logica sandbox EHR environment using the Substitutable Medical Applications and Reusable Technologies (SMART) guidelines on Fast Health Interoperability Resources (FHIR) standards [29]. Details on the technical architecture including interoperability approach are available elsewhere [30]. Physician-patient dyads participated in a simulated virtual encounter using an online meeting software and provided consent prior to starting the simulated encounter. We instructed both physicians and patients to conduct themselves as they would in an encounter where an anticoagulated patient wants to use ibuprofen to treat their pain but has concerns about bleeding. Participants permitted us to record the encounters. Physicians and patients received instructions separately via breakout rooms and asked study investigators any questions prior to the simulated visit.

Physicians randomized to the DDInteract group received a link to a video demonstrating the functionality of DDInteract, while those randomized to the UC group received information on the main topic and of online drug information sources that can screen for drug interactions (see Table S1 in the Multimedia Appendix 1). Physicians in both groups received access to online resources for DDIs to use during the encounter at their discretion. After the simulated encounters, both physicians and patients participated in a semistructured interview with a member of the research team to discuss the encounter and use of DDInteract and online drug information resources. Upon concluding the simulated encounter, participants completed an online survey containing the instruments described in the sequent section.

The primary outcome was quality of SDM according to the 9-item Shared Decision-Making Questionnaire (SDM-Q-9) questionnaire assessed by physicians. The physician postsimulation survey included items in 6 groupings: (1) SDM-Q-9 scale [31]; (2) adapted System Usability Scale (SUS) [32]; (3) National Aeronautics and Space Administration (NASA) Task Load Index (NASA-TLX) [33]; (4) Unified Theory of Acceptance and Use of Technology (UTAUT) [34], including the performance expectancy, effort expectancy, attitude toward using technology, social influence, facilitating conditions, self-efficacy, anxiety, and behavior constructs; (5) Perceived Behavioral Control (PBC) scale [35]; and (6) questions pertaining to participants’ demographics and professional experience. The patients’ survey included 6 items adapted from the SUS scale, 11 items adapted from the Decisional Conflict Scale (DCS) [36], and questions about their demographics and anticoagulation treatment.

We calculated the mean and standard deviation for each item/construct from the various scales and instruments. For SDM quality, which was the primary outcome, we conducted a 2-group Student t-test assessing differences for DDInteract versus UC as rated by physicians. A similar approach was used to test for differences in secondary measures including the SUS, NASA-TLX, and UTAUT. The PBC scale rated by physicians was also analyzed using the same approach, as well as items on the DCS. The total time of the simulated encounter for both groups was compared using Student t-test. All statistical comparisons used a 2-tailed test with an alpha <.05. Given the relatively small sample size, no adjustments were made for multiple comparisons. Since we could not assume that the populations were normally distributed, we estimated the Mann-Whitney U test for all the outcomes results reported. We also reviewed transcriptions of the post encounter semistructured interviews and highlighted the most insightful comments. All transcriptions were reviewed, and the principal investigators extracted the most relevant and repeated comments. Data were analyzed using Microsoft Excel for Mac 2016 (IBM Corp) and Stata (version 17, basic edition).

A total of 12 physician-patient dyads completed the formative evaluation. We randomized 6 dyads to DDInteract and 6 to UC. Participant demographics appear in Table 1. The UC group had 1 more female physician and 2 more female patients than the DDInteract group. All patients and 9 (75%) of the physicians reported being Caucasian. There was no difference in physicians’ self-reported experience managing patients with warfarin between the 2 groups (DDInteract 68.8, SD 31.5 vs UC 77.7, SD 20.6, from a 0-100 scale, P=.58).

The primary outcome of interest (SDM), as measured by the SDM-Q-9, appears in Table 2. Perceptions of SDM differed significantly between DDInteract and UC groups for 6 of the 9 attributes (P<.05) and include: facilitated the discussion on preferences, assisted patient comprehension, presented different options, presented advantages and disadvantages of the different treatment options, involved the patient, and provided clarity on the decision to be made. Not surprisingly, the largest difference between the groups was with respect to presentation of different treatment options (mean 5.7, SD 0.5 vs mean 1.8, SD 1.2, P<.001) and presentation of advantages and disadvantages (mean 5.0, SD 1.1 vs mean 2.8, SD 1.2, P=.008) for DDI and UC, respectively.

The task load assessed by the NASA-TLX index, scaled from zero (low/easy/successful) to 10 (high/demanding/unsuccessful), was lower for the DDInteract group than the UC group; it was not statistically significant but trended toward significance (mean 2.0, SD 1.2 vs mean 4.2, SD 2.1, P=.08, respectively) (Table 3). Participants indicated that DDInteract took significantly less effort to use than the tools in UC (mean 0.8, SD 0.7 vs mean 7.3, SD 3.7, P=.008, respectively), but none of the other constructs differed significantly between the study groups.

Patient-reported scores on the adapted SUS scale did not differ between groups, but physician ratings differed significantly, with DDInteract perceived as more logical, efficient, and helpful/effective than UC and SDM perceived as more valuable than using traditional DDI tools in UC (P<.05) (Table 4).

Physicians’ responses on a Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree) in the DDInteract group differed from those in the UC group on the following UTAUT constructs: performance expectancy (mean 5.6, SD 1.2 vs mean 3.4, SD 1.7, P<.001), effort expectancy (mean 6.8, SD 0.5 vs mean 5.2, SD 1.6, P<.001), attitude toward using technology (mean 5.7, SD 1.1 vs mean 3.9, SD 1.8, P<.001), social influence (mean 5.5, SD 0.8 vs mean 4.1, SD 1.8, P<.001), and anxiety (mean 1.4, SD 0.5 vs mean 2.5, SD 1.5, P<.001), respectively (Table 5). In general, it appears that physicians perceived DDInteract to perform better, require less effort, improve attitudes toward technology, be supported by administration and colleagues, and reduce perceived anxiety more compared to UC.

Table 6 shows the physicians’ assessment of PBC. Physicians exposed to DDInteract were more likely to indicate using SDM in the clinic compared to physicians randomized to UC (mean 6.8, SD 0.4 vs mean 6.0, SD 0.6, P=.02, respectively). DDInteract physicians were also more likely to perceive conducting SDM without extending the duration of the visit than UC physicians (mean 5.5, SD 0.5 vs mean 3.7, SD 1.5, P=.03). However, the mean duration of the encounters increased 5 minutes on average for the DDInteract, but this increase did not significantly differ between the DDInteract and UC groups (mean 17.6, SD 5.4 minutes vs mean 12.7, SD 4.8 minutes, P=.13). No other PBC items differed significantly between the 2 groups. There were no differences between the groups in patients’ ratings on the DCS (Table S2 in Multimedia Appendix 1).

The results of the nonparametric tests for the different outcomes (SDM-Q-9, UTAUT, SUS, NASA-TLX, PBC, and DCS) did not differ in terms of statistical significance from the ones presented above (Tables S3-S9 in Multimedia Appendix 1). Qualitative narrative data were collected to gain insight into participants thoughts on SDM and DDI.

After the virtual encounters, both clinicians and patients discussed the simulation using a semistructured interview format to elicit their impressions of DDInteract and online DDI resources. Below are select quotes from the participants. In general, DDInteract was well received by the physicians, as indicated by the following statements:

Concerns about extra time involved in using DDInteract were not evident among the participants. One physician stated:

Another physician felt that the tool would encourage more conversations with the patient and help them understand the risks associated with bleeding and anticoagulation treatment:

When patients were asked about the icon array in DDInteract as an approach to display the risk of bleeding, some comments included:

Table S10 in Multimedia Appendix 1 provides additional comments made by the physicians and patients.

Results of the formative evaluation suggest that there was a positive opinion of DDInteract as a potentially useful tool to help facilitate SDM concerning concomitant therapies that could result in a DDI. Physicians perceived the tool as intuitive, easy to use, and not increasing the amount of time during an encounter. After the simulated encounter, patients exposed to the DDInteract tool commented that they liked the way the information was presented, including the quantitative estimation of the risk and how risk changed according to patient factors in the user interface.

Physicians that used the DDInteract reported enhanced SDM compared to physicians randomized to UC using a standardized and validated tool (SDM-Q-9) [31,37]. This study demonstrates that DDInteract assisted in facilitating discussion, increasing patient comprehension, providing different treatment options with their respective advantages and disadvantages, and providing clarity around the treatment choices and the final decision. This contrasts with current warnings about potential interactions that are clinician orientated and not designed to be shared with patients.

Due to the ubiquitous use of NSAID pain relievers, including both prescription and over-the-counter products, patients play an important role in preventing harm from DDIs. Through education, patients can reduce harm by avoiding interacting OTC medications and monitoring signs and symptoms of adverse consequences [38]. One of the most important aspects of DDInteract is the risk array, which dynamically changes when risk factors or medications are selected. Usability studies have demonstrated that this approach is appealing and helpful for displaying the risk of harm [25]. Furthermore, physicians using DDInteract during the encounter received no additional training in SDM but rated SDM aspects higher than physicians randomized to UC.

Results from the SUS index suggest that physicians rated DDInteract as more logical, efficient, valuable, easy to use, and helpful/effective than physicians in the UC who rated online compendia and traditional EHR tools. This finding is not surprising given the lack of patient-specific information provided by both EHR systems and online resources for DDIs. Only 1 attribute of the SUS (“learned something”) did not differ significantly between groups (P=.06). Patient perceptions of DDInteract and UC tools did not differ on the SUS scale, a finding that might be attributable to patients wanting more information about DDIs [39,40].

DDInteract appeared to be intuitive and easy to use according to participant ratings on the NASA-TLX and UTAUT instruments. These results support those of a previously reported usability study that was conducted when designing the tool [25]. The NASA-TLX has been previously used to evaluate the workload associated with the new Clinical Decision Support Tool similar to DDInteract [41]. Compared to UC, physicians rated DDInteract higher on the UTAUT domains of performance expectancy, effort expectancy, attitude toward the technology, and social influence. Furthermore, DDInteract was associated with lower anxiety than UC resources for DDIs. These findings align with a metaregression of studies that evaluated effective clinical decision support using the UTAUT model and suggested that effort expectancy, facilitating conditions, and performance expectancy had a significant impact on clinician behavior [42]. Numerous studies evaluated medication-related CDS using the UTAUT, including studies examining a comprehensive CDS suite of tools called Sentri7/Quantifi [43], use of a decision aid for psychotic disorders [44], medication management in oncology settings [45], and a tool to help remind HIV positive men to use antiviral therapies when undertaking sexual activities [46]. However, to our knowledge, none have applied the UTAUT model to DDI-specific SDM.

Patient responses to the DCS did not differ between groups, which is unsurprising. Studies using the DCS in the context of nonvalvular atrial fibrillation (NVAF) found mixed results with patient decision aids [47,48]. NVAF decision aids focused on anticoagulant treatments show lower DCS scores or a decrease in the DCS score after using the decision aid, but these differences were not significantly different when compared with groups not using such aids [18,47,48]. In addition, the small sample size may contribute to the lack of significant differences between the groups with respect to the DCS in our study.

The time required to use CDS and SDM applications is a common concern among clinicians. Our study found that physicians did not perceive DDInteract to affect the amount of time or effort spent with patients (PBC scale question on performing SDM without extending the time of consultation favored DDInteract, P=.03), though there was a nonsignificant 5-minute difference in the time to conduct the simulated encounter, with DDInteract encounters being longer. During the debrief, 1 physician proposed that DDInteract could actually save time because it visually explains the risk of harm and various treatment options, as compared to having to explain the risks verbally. This disconnect between observed time to use the DDInteract and perceived time to use the tool is probably due to clinicians, who had no previous experience with the tool, having to learn how to use DDInteract during the simulation. Clinicians found the tool easy to use and intuitive, indicating a reduced time to use the tool in future encounters.

DDInteract is a novel approach to providing information about potential interactions. Currently, online drug compendia and warnings within EHR systems have numerous issues [49]. They do not account for individual patient risk factors or provide quantitative estimates of the risk. Additionally, they lack the ability to explore changes in the risk with therapeutic alternatives and are almost completely text based without any visual display of information. Furthermore, they may contain incorrect information. For example, Drugs.com currently states that the mechanism to control bleeding risk in patients on warfarin and ibuprofen is to closely monitor the international normalized ratio (INR); however, concurrent use of NSAIDs and warfarin interaction is a pharmacodynamic interaction that will not affect the INR [24]. Studies have found patients’ INRs to be within range among individuals experiencing bleeding and taking both medications, and a systematic review did not find an effect on INR for ibuprofen or naproxen [50,51]. A review of 7 Chinese apps that support DDI checking and target health care professionals had tremendous variation in information about DDIs, with over 50% lacking accurate information on DDIs [52]. Although DDInteract is currently limited in scope in terms of the interactions it includes, the approach is generalizable to other interactions, and efforts are underway to expand these interactions.

This study has several limitations that should be considered when interpreting the results. We recruited fewer participants than our a priori sample size (16 dyads), but we still observed significant differences between the study groups in terms of the primary outcome of SDM quality and several secondary outcomes of interest. COVID-19 protocols and restrictions limited our recruitment of subjects because clinic visits have been in part shifted to telehealth, making it more difficult to contact patients, and because physicians were overburdened with demands due to the pandemic [53,54]. In addition, this study focused on a specific DDI involving warfarin, which is being prescribed less because of alternative anticoagulants that require less frequent monitoring. Several physicians commented that they would have liked to see DDInteract incorporate other anticoagulants and other bleeding risks beyond GI. As stated previously, efforts are underway to broaden the interactions incorporated into the tool. Another limitation is that the clinical scenario required dyads to role-play a scenario that may not occur in clinical practice, where a patient is seeking advice from the physician about treating pain while on an anticoagulant. While this scenario may be infrequent in most practice settings, previous studies have found this to occur in up to 25% of patients receiving warfarin [2,3]. Furthermore, many physicians in the study indicated they were aware of the interaction when researchers explained the scenario prior to the simulated visit. Thus, many practitioners may have behaved differently when provided with DDInteract or other DDI information. Nonetheless, after the simulated visit, many practitioners exposed to DDInteract were enthusiastic about the tool. Therefore, we believe DDInteract may be useful in many practice settings, especially as the tool is expanded to support SDM for other drug interactions, such as other oral anticoagulants.

DDInteract is a novel tool designed to facilitate SDM during encounters concerning potentially harmful DDIs. The tool improved SDM, was not overly burdensome according to the NASA-TLX and UTAUT scales, and was generally well received by both physicians and patients. However, further research is needed to evaluate the impact of DDInteract on exposure to harmful DDIs and other important health outcomes.