This is a subgroup analysis of the Cobalt+ RCT, a separately reported RCT [16]. The parent RCT randomized 1275 HCWs to test the effect of proactive digital mental health messaging (Cobalt+). A total of 633 of the HCWs were randomized to usual care, and 642 HCWs were randomized to Cobalt+. Eligible HCWs were employed by a large, urban, academic health system in the Northeast and provided informed consent. Participants were excluded if they were non–English speaking, did not have daily access to a smartphone, or did not participate in at least 4 hours per week of patient care. Recruitment was conducted via email from January to May 2022. Eligible and consenting individuals were enrolled and randomized on a rolling basis, and the final participant completed follow-up in March 2023 [16].

Cobalt+ intervention participants received a suite of services via proactive text messaging, including (1) monthly automated text messaging reminders and links to Cobalt, and (2) bimonthly text-message–based measures of depression and anxiety. Participants included in this analysis are those randomized to Cobalt+ and who visited the Cobalt platform due to the text messages and completed usability and acceptability measures. The trial was approved by the institutional review board at the University of Pennsylvania (IRB protocol #: 848844) and preregistered on ClinicalTrials.gov (identifier: NCT05028075).

The intervention and data collection process was entirely remote. Outcome measures and text messages were administered in English and electronically using a Health Insurance Portability and Accountability Act (HIPAA)–compliant research platform to facilitate remote engagement for clinical trials and care [17].

All participants completed baseline demographics before randomization; variables included age, sex, race, ethnicity, marital status, and professional role. They also completed depression (Patient Health Questionnaire-9 [PHQ-9] [18]) and anxiety assessments (Generalized Anxiety Disorder-7 [GAD-7] [19]), both of which are widely used and are validated instruments and have been used in multiple studies and efficiently screen for symptoms of depression and anxiety. All participants completed PHQ-9 and GAD-7 follow-up assessments at months 6 and 9. Usability and acceptability outcome measures for this analysis were collected cross-sectionally at the 6-month survey only. The datasets generated or analyzed during this study are not publicly available due to participant privacy, but a deidentified dataset is available from the corresponding author on reasonable request. Of note, the PHQ-9 and GAD-7 data are available for the public to access through the National Institute of Mental Health Data Archive, which makes data available across studies from a variety of scientific domains.

Descriptive summary statistics were used for usability (SUS) and acceptability (NPS) outcome measures, and a chi-square test was used to detect group differences (Stata SE 17, Stata Corp LLC). All available data were analyzed; no additional procedures for handling missingness were undertaken.

Open-ended questions were analyzed using a qualitative open coding process and a general inductive approach. Two trained coders (MS and LS) jointly read 50 responses and drafted the outline of the codebook together. The 2 coders self-identified as female, and one as White and one as South Asian. Each coder had 10+ years of experience conducting and analyzing qualitative data. Each coder then independently read 50 more responses to adjust the codebook as needed. The coders met in person every 100 responses to resolve discrepancies and finalize coding procedures. If consensus was not met during these sessions, a senior research team member and qualitative researcher (AA) adjudicated. Coding decisions and codebook revisions were documented in an audit trail to enhance transparency and rigor. Following adjudication of any areas of discrepancy, the coders shared an additional 10 responses to review coding for interrater reliability and overall comprehensiveness. Following adjudication of any areas of discrepancy, coders independently coded the remaining responses. The coders used Microsoft Excel (Version 16.91; Microsoft Corp) to calculate interrater agreement (Cohen κ above 0.80). Strategies used to ensure reliability and validity in the qualitative data included a comprehensive audit trail, checks between coders, and biweekly team debriefings.

The trial was approved by the institutional review board at the University of Pennsylvania (IRB protocol #: 848844), preregistered on ClinicalTrials.gov (Identifier: NCT05028075). Written informed consent was obtained from all participants. A safety protocol was approved by the institutional review board and overseen by a 3-member independent data and safety monitoring board. Participants reporting suicidal ideation through surveys or text messages were promptly contacted by the University Employee Assistance Program (EAP) for a safety assessment, with up to three outreach attempts within 3 days. Automated safety messages with crisis resources were triggered by predetermined self-harm–related keywords. All participants received information about national suicide prevention resources and mental health support at study completion. Participants received compensation for completing the baseline survey ($50), 6-month survey ($100), and 9-month survey ($50) via electronic gift card. The study was conducted in accordance with the Declaration of Helsinki and local regulations. Participant data were deidentified and stored on encrypted servers.

Of the 642 HCWs randomized to the intervention. Of the 642 Cobalt+ participants, 25.2% (162/642) visited the Cobalt platform due to the proactive text messaging, completed usability and acceptability outcomes, and were included in this analysis. Among the 162 participants who completed the outcomes, the mean age was 38.9 (SD 10.5) years, with nearly half (48.2%) aged 18-35 years. The majority were female (147/162, 90.7%), 56.8% White (92/162), 53.1% (86/162) married or living with a partner and approximately one-third engaged in shift work (56/162, 34.6%). In terms of professional roles, 27.8% were nurses, 12.4% were physicians or advanced practice providers, and 25.9% were in managerial positions. Baseline mental health symptom scores were mild, with mean PHQ-9 and GAD-7 scores of 6.5 (SD 5.3) and 6.4 (SD 4.9), respectively. There were no significant differences in terms of demographics and baseline mental health measures (eg, PHQ-9 and GAD-7) between those who completed the assessments and the overall study sample, except for gender (χ²₆₄₀=8.86; P=.00), with a higher proportion of female participants and managers (χ²₆₄₀=6.58; P=.04) who completed the outcome measures (Multimedia Appendix 1).

The results of the primary RCT demonstrated improved mental health outcomes among those HCWs randomized to Cobalt+, a proactive service sent via text messaging, including (1) monthly automated text messaging reminders and links to Cobalt, and (2) bimonthly text-message-based measures of depression and anxiety [16] This analysis investigates how usable and acceptable the proactive support approach was to HCWs and offers 2 main findings. First, Cobalt+ had “good” usability (SUS score=74.43). Previous literature notes that an SUS score above 68 is above average, while scores between 70 and 84 are considered “good usability” [20]. In addition to the SUS composite score, descriptive statistics for individual SUS items were provided to illustrate response patterns across usability domains. When analyzing open-ended usability feedback, the most used resources were videos, podcasts, and articles. This is an important finding as videos, podcasts, and articles are largely cost-effective and can reach a broad audience simultaneously. Second, the NPS score (n=13.7) was positive and, in a range, considered to reflect moderate to low loyalty, suggesting both promise and opportunity for improvement [22]. Despite the positive NPS score, it is essential to note that our sample had a substantial proportion of detractors (52, 32%) and this proportion should not be overlooked.

Open-ended questions provided additional context about opportunities to improve the proactive approach. Lack of time, not needing therapeutic services, or having their own mental health provider were often expressed as the main reasons for not accessing resources. However, some participants enjoyed the proactive outreach and wrap-around support. This finding speaks to the value of the proactive approach where the support and resources are centered around the user. This is particularly relevant, as many digital platforms require users to seek out resources, and mental health conditions can compromise insight, motivation, and decision-making, thereby making self-directed engagement in care. This analysis highlights the importance of “quick and easy” survey assessments and how these periodic check-ins “allow [participants] the freedom to answer when it is convenient… reminds [them] when it's time to self-reflect” and how the frequent outreach made participants feel “cared for.” Yet, a smaller group found the text messaging to be intrusive. Future research could allow individuals to customize their communication type (ie, text message or email) and how often they receive notifications. Yet, this customization does not overlook implementation challenges (eg, integration across systems, resource allocation, and clinical support) [14,24], structural considerations (eg, disparities in access to digital tools), and cultural concerns (eg, stigma and trust in employer-sponsored tools outside of a research context) [25]; however, it is a first step to personalized mental health and well-being outreach, which will lay the foundation for future iterations, refinements, and adaptations to meet the HCWs’ mental health and well-being needs [26].

The digital strategy to send resources and assessments to HCWs was overall found to have good usability and was modestly received. This study reveals formative and important preliminary data that remote text message engagement can extend the reach of mental health platforms for HCWs. Proactive outreach to intermittently engage, assess, and connect health care workers allows one to prioritize mental health and well-being and motivate them using existing resources [27]. This approach also maintains privacy and provides a unique strategy at scale, which can be tailored to the needs of an individual over time.

This is among the first large mental health–focused RCTs to collect participants’ feedback and feasibility metrics among HCWs during the pandemic and beyond. Prior research found a positive impact of an employer-sponsored behavioral health program on improved depression and anxiety symptoms through increased psychotherapy sessions [28,29]. This is a noteworthy difference compared to the Cobalt+ proactive approach and study sample for 2 reasons. First, Maeng et al [28] analyzed individuals whose baseline scores were more than or equal to 10 on depression (PHQ-9) or anxiety (GAD-7) validated measures [28], whereas the Cobalt+ sample had mild baseline PHQ-9 and GAD-7 scores (Multimedia Appendix 1). Given that Cobalt+’s sample exhibited mild baseline symptoms, there was limited potential for measurable improvement in depression and anxiety. Second, despite the Cobalt+’s samples’ mild baseline scores, nearly 20% used appointments with a mental health professional (Table 1). This suggests that individuals with subthreshold forms of depression and anxiety also seek out mental health professional support.

Among digital platforms catered to HCWs, our analysis is most similar to the COVID Coach [9] and Wellness Hub [11] apps. The COVID Coach, smartphone-based app, was tested through an RCT among 40 South African physicians. It was noted to have adequate feasibility acceptability as measured by the SUS and client satisfaction questionnaire (CSQ-8). COVID Coach’s SUS score was slightly higher than Cobalt+’s with 76.6 (SD 14.6). Our analysis did not use the CSQ-8 since Cobalt+ provided a variety of tools and resources and was not limited to mental health services. Nevertheless, the authors recommend future in-depth qualitative interviews to capture users’ experience with the app and patterns of use [9]. In its design and implementation, Cobalt+ also closely parallels the Wellness Hub app [11], sharing similarities in its resource offering and being deployed within a large health system in the Northeast United States. However, to our knowledge, Wellness Hub has not been extensively evaluated. Golden et al [11], reports back-end metrics such as the number of HCWs downloaded the app, the total number of baseline surveys submitted, the number of users submitting additional surveys postbaseline, and general app engagement metrics (eg, the number of app openings). Similarly, their evaluation is lacking the user experience of the app [11].

This analysis has several limitations. First, not all study participants (24.6%) completed the month 6 assessment, and fewer completed the outcome measures (n=162). This limits our generalizability in reflecting the larger participant experience due to the limited responses. Usability and acceptability findings should be interpreted carefully due to this response bias. Second, study participants were recruited from one large, urban health system, and second, the inclusion criteria required participants to have daily access to a smartphone; this criterion may limit generalizability, especially within the context of the known racial and ethnic “digital divide.” This divide may be smaller in employed health care workers, and smartphone ownership continues to expand in the United States [30]. Third, a higher proportion of female participants completed the outcome than the overall study sample measures (Multimedia Appendix 1). Interpretation of our findings and external validity may be limited for dissimilar HCW samples. Fourth, study participants were HCWs willing to complete 2 online surveys; therefore, selection and nonresponder bias may be present. Fifth, participants who opted into the study may have experienced greater distress and were likelier to prefer mobile interventions than those who declined participation. Sixth, validated survey measures lack some specification. For instance, the SUS provides a subjective, global measure of usability and unfortunately does not specify specific domains such as navigation, error prevention, and efficiency. Similarly, despite the NPS’s widespread use, the acceptability thresholds are derived mainly from commercial rather than health care contexts [22]. Our analysis reported SUS and NPS scores and open-ended feedback separately. Seventh, there remains stigma associated with accessing mental health care, which may also contribute to selection bias and limited generalizability. Rather than integrating mental health records or treatments, we relied on self-reported data from participants and data on appointments made. Self-report measures have limitations due to social desirability and recall errors [31,32]. Further research is needed to continuously seek feedback from those using the program to stay relevant and responsive to ongoing and changing stressors.

Proactive mental health services may help overcome expected barriers to uptake of passively available services. This study demonstrated that Cobalt+, a proactive digital mental health program using proactive text messaging, is modestly usable and acceptable based on two key evaluation metrics, further supplemented by HCW feedback. These findings suggest the program's viability and the need for additional research toward improvement and broader implementation. In this study, we report on the insights gained using simple, scalable text messaging to reach a large group of diverse HCWs. Future studies are needed to test this novel approach across clinical settings and professional roles and in large health systems.