Members of the primary research team led earlier engagement efforts that informed MS CATCH development, namely, creating the BRIDGE dashboard concept [21], testing the acceptability of patient-reported outcomes (PROs) in a point-of-care MS clinical dashboard (MS NeuroShare; 19]), and developing a closed-loop reporting system that collects and synthesizes data at the point of care (MS Falls Insight Track; MS-FIT); [22]). Each of these design processes informed the current technical build.

BRIDGE [21] is a technologically scalable, institutionally approved, workflow friendly, cross-disease, modular precision medicine platform [23]. BRIDGE launches from within a patient’s encounter from the Epic electronic health record (EHR) at the University of California, San Francisco (UCSF) using industry standard integration (Figure 1; Table S1 in Multimedia Appendix 1 [24]), delivering a seamless experience for clinicians [21]. BRIDGE was designed using an extensive process of HCD [19,20]. The tool accesses and visualizes data that are currently available in various disparate sources (EHR, patient diaries, and publicly accessible websites), with the substantial innovation that they are presented here in one comprehensive and streamlined format. A disease-specific version of BRIDGE is live within multiple clinics at the UCSF, with the MS clinic BRIDGE used as a point-of-care dashboard for the current project.

This closed-loop prototype designed for a collaborative health system integrated a patient-facing PRO app with an EHR-based clinician-facing dashboard [25]. Patient participants cited the perceived value of thinking about and recording information before their appointments, noting how it impacted discussions with their clinicians, adding that coviewing information with the physician put physicians and patients on the same page and promoted a conversation of equals. Clinicians perceived patients’ prospectively collected mood scores as more sensitive, accurate, and comprehensive than a patient’s recall during the visit, and felt that these could improve value and promote clinician engagement [25].

A closed-loop system to prospectively record, report, and prevent falls in people living with MS was designed using HCD in a process analogous to that intended for MS CATCH [22]. MS-FIT includes low-burden regular falls ascertainment that patients can access with one click from any computer, tablet, or device, triggering a clinician’s inbox message for new or serious falls. Clinicians can then access and launch, from the medical record, a comprehensive version of BRIDGE refined to support multimodal falls assessment and prevention [22]. The MS-FIT development process informed the feasibility of a closed-loop approach as well as specific needs and concerns of intended users, people living with MS, and MS care teams.

The prototype for MS CATCH used in this study was refined for the specific purposes of depression monitoring and treatment by the study team using the BRIDGE and MS-FIT technical scaffolding, and informed by the MS NeuroShare and MS-FIT design processes and experiences. The MS CATCH prototype used in this study is shown in Figure S1 in Multimedia Appendix 1.

The PRO data obtained from the patient participants populate the BRIDGE dashboard in real time. Patient visits were randomized 1:1 using a simple randomization scheme in which the clinician was asked by the study team to launch BRIDGE during the participant’s visit. For those who did not use BRIDGE, the visit proceeded as a usual clinical visit. Clinicians were reminded to launch BRIDGE via text or email before the patient appointment when applicable, and neither clinician nor patient participant were blinded to the intervention. Patients continued to be recruited until a target (n=50) had met the inclusion criteria and were seen in the clinic. For participants with PHQ-9 scores of ≥19, even if their clinician was not asked to use BRIDGE, the clinician was informed of their score near the conclusion of the clinic visit to ensure safe treatment of the patient. At the conclusion of the study visit, both clinician and patient feedback regarding the tool were solicited. During the proof of concept pilot, MS CATCH was used once during the visit in real time. For the full RCT, the patient-facing mood survey will be completed monthly, as described in the results for phase III.

This study was approved by the UCSF institutional review board (#18-26148).

The participant demographics and usability outcomes were analyzed using descriptive statistics. The main efficacy outcome, percentage visits where depression was mentioned in the clinic notes, was compared for visits with and without the use of MS CATCH using chi-square analysis [33].

HCD is a process that holds at its center the needs of the intended users. During the MS CATCH refinement phase, the overall goal was to fine-tune and optimize the intervention by partnering with patient and clinician users and multidisciplinary stakeholders in a sequence of iterative feedback sessions to inform and validate design decisions (Figure 2). To our knowledge, features assembled into the MS CATCH prototype had never before been available at the point of care in a clinically actionable format for depression treatment in MS. It was important to verify that the workflow and displays promoted behavioral change likely to improve depression care and to simplify the display into the features most likely to drive adoption of the tool and support depression care. Here, the “capability, opportunity, and motivation model for behavioral change” (COM-B) [18] was used. COM-B is a hub in the center of Michie’s behavioral change wheel [18] and can help inform a patient’s intention to engage in a planned behavior, which is considered the best predictor of that behavior [34,35]. The COM-B approach, and individual intervention functions, have been successfully applied to facilitate behavioral change in people living with MS [17].

To maximize the likelihood that the tool would be adopted and effective, it was evaluated using the health IT usability evaluation model [32]. This model integrates multiple usability theories including the Technology Acceptance Model and evaluates both subjective and objective outcomes. Although refining the critical data and visualization elements, as well as technological and clinical workflow aspects, the 4 key variables proposed by Mathews et al [36] for digital health tool validation were examined to determine whether the tool (1) reflects HCD principles and (2) is likely to engage patients. These factors include usability, effectiveness, learnability or ease of use, and likeability. Usability was defined using the System Usability Scale, learnability and ease of use were measured with a subset of the Health Information Technology Usability Evaluation Scale–based questions, and likeability was assessed with a single Likert scale question: “Do you like the tool?” and the net promoter score (NPS). Effectiveness was determined based on the pilot study visit notes outcomes.

Over a 9-month period between September and June, sequential activities were conducted as outlined in Figure 2.

This group was convened and led by members of the primary research team. The primary research team comprised investigators with a track record of collaboration as well as expertise in both scientific and technological aspects of this project, including MS clinical trials (RB), digital tools to evaluate and treat cognition and mood in MS [16] (RB and AF), psychiatric care for underserved populations and innovations in treatment of psychiatric conditions in diverse settings (CM) [37-44], implementation science (CM), statistics (Ann Lazar), patient engagement (RB and JR) [20,21,25], HCD of digital tools (JR, RB, NS, and NM) [20,21,25], and launching an institutionally and technologically sophisticated, scalable, cross-disease platform from the EHR at UCSF (RB and BRIDGE team). The stakeholder advisory group included the research team listed above, as well as a patient champion JS, patient advocacy group leader Linda Glassel, National Multiple Sclerosis Society, Northern California Chapter President, MS nurse expert AM, Registered Nurse, social worker MD licensed independent clinical social worker, and MS neurologist CYG.

MS clinicians and other experts (degrees including Doctor of Medicine, Nurse Practitioner, Registered Nurse, and Masters in Social Work) were identified within the UCSF MS clinic and the investigators’ broader professional network. Adults with MS were recruited from the UCSF MS center by the study team (convenience sampling). All participants provided informed consent to test the tool and provide preliminary data. Furthermore, 31% (10/32) of the patients participated in the phase II pilot study.

Activities during this phase (HCD phase) were approved by the UCSF institutional review board (#22-36620).

Of the 200 patients contacted, 106 (53%) agreed to participate in this study. Of these, 50 (47.2%) met the inclusion criteria of a PHQ-9 score >4 and were cared for by 6 participating physicians. Distribution of PHQ-9 scores was as follows: 52% (26/50) mild depression (5-9); 32% (16/50) moderate depression (10-14); 10% (5/50) moderately severe depression (15-19); and 6% (3/50) severe depression (20-27). Patients were then randomized to visits where the BRIDGE dashboard was used (24/50, 48%) or not used (26/50, 52%). All participants were contacted before their clinical visit, which was conducted between May and December, 2021. This visit was where the MS CATCH intervention was used. The cohort included 40 (80%) females and 10 (20%) males, aged 25-75 (mean 50.46, SD 13.0) years, including individuals with all MS subtypes (32 relapsing remitting MS, 9 primary progressive MS, 1 progressive relapsing MS, 6 secondary progressive MS, 1 unspecified, and 1 MS likely).

The mean likeability score was 4.3/5 for patients (93% of the patient participants agreed or strongly agreed that they liked the tool) and 4.4/5 for clinicians (100% of the clinicians agreed or strongly agreed that they liked the tool).

For perceived usefulness [30,31], among clinicians, 100% of them reported that the tool increased their attention to the patient’s mood, particularly for patients with mild to moderate symptoms that might otherwise have been missed. Clinicians provided some specific feedback on how to further customize the dashboard, suggesting that it would be better to have the categorical components of the PHQ-9 rather than just the raw score (3/6, 50%).

Among patients, 71% (20/28) reported feeling that it was useful to report their mood-related symptoms before the visit. In qualitative feedback, many patients reported that filling out the mood survey ahead of time gave them the opportunity to reflect on their own feelings: “It set the tone, upfront, about how I was feeling,” “It’s helpful for me to think about assessing my mood in a structured way,” “was useful because it made me stop and reflect,” and “it opened the door to explain how...new MS symptoms are affecting my mood.”

For the patients with a PHQ-9 score of ≥5, visit notes documented a discussion of depressive symptoms in 75% of the visits where MS CATCH was used versus 35% of the visits where it was not used (N=50; χ²₁=8.2; P=.004; Table 1). One patient was referred to a psychiatrist for suicidal ideation.

Overall, 45 individuals provided feedback on the tool during successive phases of development. The results are summarized in Table 2.

Detailed findings and scores from each round of engagement, as well as iterative development steps, are presented in Multimedia Appendix 1 and summarized here. During the discovery interviews (N=5 patients, N=5 clinicians), each tool component was found to be useful and usable. A number of features and data types were identified that mapped to the COM-B principles of behavior change, and these resulted in design changes and interventions promoted to address potential boosters or blockers (Table S2 in Multimedia Appendix 1). For example, patients universally felt that the PHQ-9 should be delivered monthly if not more often, despite concerns by the study team that it would be too burdensome. The patient-facing survey was edited to support free-text entry and addition of language at end of survey informing patients who scored >15 to seek care urgently, as their care team may not see the score for 36 hours, and instructing patients to seek immediate care if experiencing a mental health crisis. During the think-aloud sessions (N=7 patients, N=5 clinicians), some patients articulated concern for how answers might be interpreted by their clinicians, for example, concern about sending up unnecessary “red flags” with their survey answers as symptoms addressed on the survey can overlap with MS symptoms. Relatedly, the free text box proved to be a popular design feature as it presents an opportunity to offer context for potentially concerning or MS symptom-related survey answers. Additional features were noted to be of value; for example, neighborhood walkability was specifically identified by patients as useful context for their clinicians to have within the clinical decision support view. Patients also universally expressed interest in having dashboard views available in a summary display for reference following appointments. All surveyed participants preferred an electronic summary format distributed through the patient portal (MyChart) as opposed to an alternative format that was not integrated with their EHR portal. Clinician feedback led to a reduction in the visual “wordiness” of the original clinical decision support widget. One decision that could not be accommodated was the ability to include patient insurance information, which substantially influences prescribing choices and referral decisions, as there is unfortunately no “ground truth” source for this information.

The patient survey was further developed to include an in-survey pop-up directing patients who screened positive for suicidal intent to seek immediate medical care (with contact information), a catalog of mental health resources at the end, and a companion patient-facing mood tracker displaying the patient’s longitudinal mood survey results. On the dashboard side, the clinical decision support screen was further refined to be less “visually overwhelming.”

Clinicians’ qualitative feedback on the potential impact of MS CATCH on the care delivery experience and quality of conversations with patients and caregivers ranged from appreciation for the intervention’s holistic approach to concern that limited visit times will constrain the dashboards’ seemingly limitless potential.

The final prototype is displayed in Figure 1. The patient-facing survey is easily accessible via email. Along with receiving the short monthly survey via email, the patient’s response history and resources relevant to depression self-management can be viewed at the end of the survey (Figure S2 in Multimedia Appendix 1). PHQ-9 scores of 15 or higher or any suicidal intent trigger an alert to the patient’s MS clinician’s EHR in-basket. This is a critical feature allowing for timely follow-up within the usual care workflow, including an urgent referral to psychiatry. The clinician-facing dashboard (Figure 1) has a comprehensive mood evaluation and treatment dashboard that launches from the EHR. This dashboard can be copied and pasted into the clinical notes as well as the patient’s after visit summary to be accessed at the end of the visit and between visits. The tool uses a closed loop system, diagramed in Figure 1, which integrates the patient survey, in-basket messaging, and clinical dashboard tools.

A convenience sample of adults with MS (n=20) and clinicians (n=10) participated in the final tool testing. Of the participating clinicians, 5 (50%) practiced at external (non–UCSF-affiliated) institutions, and 5 (50%) were affiliated with UCSF. The target and achieved results are presented in Table 3.

Patient assessment ratings exceeded the goals in approximately all categories. Despite a patient NPS of 50, which is considered one point below excellent, slightly fewer (75%) than the goal of 80% agreed or strongly agreed that the tool was likeable (mean 4.25, SD 1.12). Of those (n=5) who scored the tool ≤4 with regard to likeability, approximately all (n=4, 80%) were neutral (score=3). Reasons varied from lack of personal applicability, “Mood symptoms are due to MS, not depression,” and “I have a lot of anxiety, but not depression”; to survey design, “I don’t like the multiple-choice options”; and to formatting, “Font is small.” Concrete follow-on actions taken by the development team in response to this feedback were as follows:

Similarly, clinician ratings exceeded the goals in approximately all categories. A clinician NPS of 40 indicated strong favorability in terms of recommending to peers. Perceived usefulness was slightly under goal at 95%. Although all but 1 clinician agreed or strongly agreed that the tool was useful (mean 4.35, SD 0.75) and 1 rated the tool ≤4 citing concerns about limited bandwidth and lack of financial incentives or reimbursable mechanisms for clinicians to address mood between clinical encounters. This feedback echoes concerns noted during discovery interviews about visit time limitations and the need for complementary workflows to ensure that patient mood issues can be monitored in a timely fashion, with the assistance of multidisciplinary staff (registered nurse and masters in social work) and elevated to physicians as needed.