Patient and Health Care Professional Perspectives About Referral, Self-Reported Use, and Perceived Importance of Digital Mental Health App Attributes in a Diverse Integrated Health System: Cross-Sectional Survey Study

Introduction

Background

Anxiety and depressive disorders are common mental health conditions in the United States that cause functional impairment and represent a leading cause of disability [1,2]. Although evidence-based psychotherapy and pharmacotherapy for these conditions exist, the combined shortage of trained mental health clinicians and high demand for mental and behavioral health services have limited access to evidence-based care [3,4]. Thus, there is an urgent need for efficient and effective treatments that are innovative, scalable, and sustainable to address the growing public health dilemma of access to evidence-based behavioral health care services.

Digital mental health applications (DMHAs) are an emerging, novel solution to address this care gap. Many DMHAs apply evidence-based principles and have accumulated empirical support primarily in controlled settings [5-7]. These DMHAs have demonstrated efficacy to reduce mental health symptoms (eg, anxiety and depression), support emotional well-being (eg, stress reduction), and improve resiliency. A recent meta-analysis of 36 randomized clinical trials comparing stand-alone DMHA interventions to non-DMHA control groups found modest therapeutic effect sizes for anxiety (Hedges g=0.31) and depression (Hedges g=0.35). Key moderators included the duration of the intervention, the presence of symptoms at the start of the study, and the types of outcome measures used [7].

While DMHAs are not a substitute for in-person care, they can be broadly disseminated to at-risk individuals to address subclinical mental health problems before clinical intervention. A stepped-care approach could reduce the demand for the number of patients requiring in-person care for mental health conditions, which would increase the accessibility and availability of in-person health care professionals (HCPs) within health care systems for persons with more immediate or serious need. Beyond this, DMHAs can serve in an adjunctive capacity to in-person evidence-based care within a new and emerging mental health care paradigm.

Digital Mental Health Integration Into Clinical Care

DMHAs hold considerable potential in complementing accessibility to traditional mental health services. Large health care systems such as Kaiser Permanente (KP) have integrated the use of DMHAs into the management of mental health conditions [8]. Beginning in 2017, KP developed a framework for implementation of DMHAs. As deployed, the DMHAs studied in this research were selected and intended to be used for self-care as part of a comprehensive ecosystem of services for improving emotional health and well-being. Using a human-centered design approach, KP assessed both clinician and patient insights to design a digital health ecosystem [8]. Since then, KP has engaged in ongoing feasibility assessment, implementation, and expansion across its regions and has previously been reported in the literature [8]. Current understanding suggests a need to further explore and refine strategies for implementation and to identify ways to best align candidate patients with the appropriate treatment for a more unified referral and patient experience.

In brief, Kaiser Permanente Mid-Atlantic States (KPMAS) is a large, diverse integrated health care system that provides comprehensive health care services to >800,000 actively insured members in Maryland, the District of Columbia, and Northern Virginia. Health care encounters across the entire care continuum are documented in an electronic health record for all members. All patients in this study were referred to at least 1 of 6 approved DMHAs, recommended by the previously described implementation framework [8], during an in-person or telehealth encounter in outpatient or ambulatory care settings by either a primary care physician, specialty care physician, or a behavioral HCP (physician or nonphysician). HCP workflow was designed to facilitate DMHA referrals using a standardized order set in the electronic health record. Continuing education, training, and individual support where necessary were offered to HCPs at various times during the implementation period. An internal website for HCPs to guide referral of the DMHAs was made available within the workflow. During implementation, behavioral health providers were authorized to refer patients for all 6 DMHAs with no registration cost. However, for primary and specialty care providers, referrals were limited to only 2 DMHAs.

After the initial rollout in December 2019, the KPMAS region observed steep increases in DMHA referrals during the first quarter of calendar years 2020 and 2021, with declining patterns in the periods between those quarters [9]. During this 2-year study period, there were variations in patient, clinical, and encounter type characteristics observed depending on whether the DMHA referrals included mindfulness and meditation alone, cognitive behavioral therapy alone, or a combination of both, and these results have previously been reported in the literature [9].

Objectives

Optimal use of DMHAs requires both clinicians’ willingness to prescribe to patients and patients’ willingness to adopt and engage in their use. With limited information available about factors that influence the adoption and use of DMHAs in routine clinical practice, we aimed to explore patient and HCP perspectives that may affect referral, adoption, and use of DMHAs for improving emotional well-being and addressing subclinical, mild to moderate mental health conditions in real-world settings.

Methods

Design

A pilot study using cross-sectional surveys spanning a 4-month period (March 2022 to June 2022) was conducted to gather patient and HCP perceptions about their recent past end user experience with DMHAs. The overarching approach for this study was guided, in part, by diffusion of innovation theory [10], where adoption of a new device, practice, or idea is dependent on three factors: (1) perception of the innovation, (2) characteristics of those who may adopt the change, and (3) contextual factors.

Where applicable, the elements of cross-sectional survey design, institutional review board (IRB) considerations, survey development and pretesting, recruitment process and sample description, survey administration, response rates, and analysis as outlined in published guidance [11] were followed and are described herein. Both the patient and HCP survey instruments (Multimedia Appendices 1 and 2) are provided for the reader to assess content, length, and structure. Because patients were explicitly recruited, completed a consent form, and were sent a participant-specific survey link, duplicate entries were avoided.

Ethical Considerations

The study was approved by KPMAS IRB (1761101). Survey responses were not linked to the electronic health record. A US $25 electronic gift card for patient participants and a US $50 electronic gift card for HCP participants were offered as an incentive if the surveys were completed. Participant-specific survey links were sent via email addresses but were stored independently from the final analytic survey dataset. Participant survey responses were not linked to gift card distribution. No personally identifiable information is included in the manuscript or Multimedia Appendices 1 and 2. All study data were stored on an internal, password-protected KPMAS server, which required dual authentication for access.

Patient and HCP Samples

To identify an eligible pool of participants, KP HealthConnect electronic health records were used to identify adult patients (aged ≥18 y) who were referred to ≥1 DMHAs between April 2021 and December 2021 during a health care encounter. This time frame for evaluation was selected to not overlap with past internal evaluation efforts. All HCPs who had referred (ie, ordered) ≥1 DMHAs during a health care encounter between the implementation of the DMHA initiative (December 2019) and December 2021 comprised the eligible HCP pool of participants.

From the eligible participant pools, quota samples of 60 patients and 12 HCPs were set as a goal for this initial pilot work. For patients, a randomized block sampling design was used with repeated sampling blocks of 63 until the quota was fulfilled. The patient sample was stratified by each referral month (April 2021 through December 2021) to ensure representation across the entire observation period and include varying durations of potential use. The HCP sample was stratified to ensure representation from physicians and nonphysicians, and primary care, specialty, and behavioral health care HCPs in randomized blocks of 30 to fulfill the quota. Participant responses were deidentified and not linked to electronic health record data to encourage participation by ensuring confidentiality of responses. For both patient and HCP samples, random selection was used to optimize representation of the eligible participation pools.

Measurements

As no standardized assessment existed due to the innovative nature of this work, clinician and patient surveys were developed de novo in accordance with accepted survey methods [12], while considering commonly accepted frameworks for technology evaluation (eg, American Psychiatric Association App Evaluation Model [13] and Technology Acceptance Model [14,15]) and other work related to the rating of mobile apps [16-18]. This was supplemented by internal evaluations from the national KP Project Chamai [8].

Once constructed, instruments were assessed for content validity by study investigators and modified via consensus discussions. Afterward, the surveys were programmed into REDCap (Research Electronic Data Capture; Vanderblit University) [19,20], which is a secure, Health Insurance Portability and Accountability Act (HIPAA)–compliant web-based application for administering electronic surveys. The instruments were iteratively refined vis-à-vis before testing within the research team and feedback from clinician colleagues. The final IRB-approved electronic versions of the instruments were distributed via email to the target patient and HCP recruitment pools derived from KP HealthConnect via participant-specific survey links.

The patient survey instrument is provided as Multimedia Appendix 1. Items were organized into the following three domains: (1) DMHA experience (ie, recall of DMHA referral from HCP, types of DMHAs used, extent of use, and reasons for initiating or continuing DMHA use; reasons for not starting or discontinuing DMHA use; and importance of DMHA attributes with respect to engagement, functionality, design, and information categories); (2) current emotional and mental health and wellness (ie, Patient Health Question 4 [PHQ-4] [21] and Subjective Happiness Scale [22]); and (3) respondent demographic information, including age, gender, education level, race or ethnicity, marital status, income sufficiency, English as a first language, general health status, health literacy [23], and confidence using a smartphone to manage emotional health and wellness.

The HCP survey instrument is provided as Multimedia Appendix 2. The questions were organized into the following three domains: (1) DMHA referrals (ie, types of DMHAs, how many DMHAs referred at 1 encounter, advantages and disadvantages of referring multiple DMHAs, and reasons for DMHA referral); (2) importance of DMHA attributes with respect to engagement, functionality, design, and information categories; and (3) respondent demographic information. Additional open-ended questions were used to allow HCPs to describe facilitators and barriers to DMHA referral and use as well as a typical patient referral to enhance discussion of the findings. Because the DMHA initiative had been implemented for >1 year at the time of survey, a description of the “typical” patient referral was requested rather than also requesting descriptions of the worst- and best-case scenarios. This approach also helped to minimize respondent burden.

For both survey instruments, questions were presented in a consistent order using a mix of yes or no, multiple selection, rank ordering, and Likert-type scale response sets. In contrast with the patient survey, open-ended questions were also used to gather a more detailed qualitative description of the HCP experience with DMHA referrals.

Procedures and Survey Implementation

Patients selected from the eligible participant pool were initially contacted via a personal telephone call and, if interested, were sent a subsequent email invitation to participate in this closed survey. The email invitation included a participant-specific link to the electronic consent form and electronic survey distributed through REDCap [19,20]. The consent form was presented in a question and answer format that included the following: study title; contact information of the principal investigator and research associate; brief study overview; why the study was being conducted; what will happen when participating; how long participation will take; potential risks, discomforts, and benefits; incentives for participation; confidentiality and safeguarding of information; reasons for withdrawal by the investigator; rights of voluntary participation; contact information for questions; and a signature block acknowledging understanding and willingness to participate. Research staff attempted to contact eligible patients by phone on a maximum of 3 occasions, with each call attempt at least 3 days apart. If a patient was not reached, a voice message was left when possible. Internal administrative policy mandates that HCPs be initially contacted by email, rather than by telephone, to invite them to participate in the electronic survey distributed via REDCap [19,20]. The survey included an initial triage question to avoid surveying patients or HCPs who did not recall receiving or providing a DMHA referral. Participants had the opportunity to review and edit their responses before submitting the survey.

For each survey invitation sent, up to 3 weekly email reminders to complete the survey were sent to interested participants, as needed. Survey links were disabled 4 weeks after the initial email was sent if they were not completed. Participants had up to 7 days to complete the survey after initially opening the link before it was disabled.

Analysis

Data were extracted from REDCap [19,20], organized and cleaned, and reported using descriptive statistics (ie, means, measures of variability, and frequencies and proportions depending on the level of measurement) to profile the patient and HCP survey responses. Rather than requiring fully completed surveys, analyses were conducted by question unless all items were necessary to calculate a specific scale result. Internal consistencies for the PHQ-4 [21] and Subjective Happiness Scale [22] were calculated to establish measurement reliability for these scales. A pairwise correlation between the PHQ-4 [21] and Subjective Happiness Scale [22] was calculated to further establish response consistency and reliability. Because each of the 4 DMHA attribute domains had a different number of attributes to rank (ie, engagement, 6/6, 100%; functionality, 5/5, 100%; design, 4/4, 100%; and information domains, 5/5, 100%), the rank-order scores within each domain were standardized to reflect a scale of 100 to simplify interpretation. To do this, individual rank-order scores for each attribute were first coded so that higher values reflected higher perceived importance. The recoded importance scores were then multiplied by (100/number of attributes in the domain) to create a scale where lower values reflected less importance and higher values reflected more importance with a ceiling value of 100. Analyses were performed using Stata (version 16.1; StataCorp LLC).

Results

Survey Response Rates: Patient Survey

There were 9525 patients referred to at least 1 DMHA between April 2021 and December 2021. Of those 9525 patients, 378 (3.97%) were randomly sampled from 6 blocks of 63 patients. A total of 119 patients agreed to participate and were sent a survey link. Of those 119 patients, 62 (52.1%) clicked on the emailed survey link and 58 (48.7%) completed the survey between March 2022 and June 2022, resulting in an effective response rate of 48.7% (58/119). Importantly, 24% (14/58) of the patients did not recall or report being referred to a DMHA and were removed from the analyses because the survey questions were intended to reflect user experience, resulting in a usable response rate of 37% (44/119).

Survey Response Rates: HCP Survey

There were 410 HCPs who had referred at least 1 DMHA during a health care encounter between the implementation of the DMHA initiative (December 2019) and December 2021. Of the 410 HCPs, 60 (14.6%) from 2 blocks of 30 HCPs per block were invited to complete the survey. A total of 12 HCPs completed the survey between March 2022 and May 2022, fulfilling the quota sample with an effective response rate of 20%.

Patient and HCP Characteristics

A total of 44 patient respondents completed the survey and reported having a DMHA referred to them by their HCP (Table 1). Of those 44 patient respondents, 32 (73%) were female participants, 32 (73%) had at least a 4-year college degree, 18 (41%) were Black or African American, and 17 (39%) were White. Mean age of the patient respondents was 42 (SD 14.08) years (29/44, 66%). Moreover, 27% (12/44) of the patient respondents screened positive for anxiety symptoms, and 23% (10/44) of the patient respondents screened positive for depression symptoms using items from the PHQ-4 [21]. Collectively, one-fourth (11/44, 25%) of the patient respondents reported a moderate to severe (≥6) PHQ-4 [21] overall score. The mean and median Subjective Happiness Scores [22] were 4.73 (SD 1.22) and 4.75 (IQR 3.75-5.75), respectively, and within the average range of 4.5 to 5.5 of other sample norms. Subjective Happiness Scores [22] were inversely correlated with PHQ-4 scores [21] (r=–0.44; P=.003; 43/44, 98%), indicating reliable survey completion.

A total of 12 HCPs completed the survey (Table 2). Mean age of the HCP respondents was 46 (SD 7.75) years (11/12, 92%), and 92% (11/12) of the HCPs were female. The mean number of years since completing training was 14.3 (SD 9.94) years (10/12, 83%). Of the 12 HCP respondents, 7 (58%) were physicians, with the remainder (n=5, 42%) either a licensed psychologist or clinical social worker. All HCP respondents personally tried at least 1 mindfulness or meditation DMHA, but only 42% (5/12) of the respondents personally tried at least 1 cognitive behavioral therapy DMHA.

DMHA Use and Referral

Approximately 73% (32/44) of the patient respondents recalled being referred to only 1 DMHA. Furthermore, 91% (40/44) of the patient respondents reported referral to 1 of the 3 mindfulness and meditation DMHAs more commonly, with only 25% (11/44) of the respondents recalling referral to 1 of the 3 cognitive behavioral DMHAs (result not reported in Table 3). Approximately 89% (39/44) of the patient respondents reported ever using at least 1 DMHA, and 61% (27/44) of the respondents reported DMHA use for ≤6 months after referral at the time of the survey. Of the 16 (36%) patient respondents who reported current use (ie, within the past 30 days), 75% (12/16) used the DMHA once daily to once weekly (result not reported in Table 3). Overall, 82% (36/44) of the patient respondents reported that DMHAs were at least somewhat helpful for improving mental and emotional health. Most common self-reported reasons for being referred to the DMHA included anxiety or panic control (25/44, 57%), mindfulness or meditation (15/44, 34%), sleep improvement (15/44, 34%), mood management (14/44, 32%), coping with specific issues such as grief and trauma (14/44, 32%), and stress reduction (13/44, 30%).

Approximately half (5/12, 42%) of the HCP respondents ordered >1 DMHA at an encounter, with mindfulness and meditation DMHAs being more commonly referred (12/12, 100%) than cognitive behavioral therapy–based DMHAs (9/12, 75%; Table 4). Open-ended text responses from HCPs revealed that patient choice, variety, and increased options were advantages to referring >1 DMHA at a time. However, HCPs recognized that this strategy may provide too much information and may be overwhelming for the patient when >1 DMHA is referred at an encounter. Most common reasons for DMHA referral included anxiety and panic control (8/12, 67%), mood management (7/12, 58%), mindfulness and meditation (6/12, 50%), stress reduction (4/12, 33%), sleep improvement (3/12, 25%), and balancing intense emotions (3/12, 25%).

Perceived Importance of DMHA Attributes

Within each DMHA attribute domain, the most important DMHA attributes that patient respondents reported (Table 5) included being fun and interesting to use (Importance Score [IS]=68.9), having personalized settings (IS=68.3), and content that appeals to personal preferences (IS=68.3) for the engagement domain; ease in learning how to use (IS=84.4) and clear menu labels, icons, and instructions (IS=69.4) for the functionality domain; visual appeal (IS=75.0) and narrator voice (IS=66.0) for the design domain; and having well-written, goal- and topic-relevant content (IS=80.0) and credible and legitimate sources (IS=67.9) for the information domain.

HCP respondents reported some similar but also some different DMHA features and ranking variability within each domain as being the most important (Table 5). They included having personalized settings (IS=75.9) and content that appeals to personal preferences (IS=75.9) for the engagement domain; ease of learning how to use (IS=94.6) and clear menu labels, icons, and instructions (IS=70.9) for the functionality domain; arrangement and size of buttons, icons, menus, and content on the screen (IS=82.5) and narrator voice (IS=67.5) for the design domain; and having well-written, goal- and topic-relevant content (IS=81.7) and credible and legitimate sources (IS=78.3) for the information domain. In open-ended text responses, HCPs also suggested other important features that included the ability to journal or take notes; having a daily mood tracker; substance use– or chemical dependency–specific application; topical content that can be upgraded; user ability to download information or audio books; and having information that is relevant and specific, rather than vague, general recommendations.

HCP-Reported Barriers to DMHA Use

From open-ended responses, the most commonly reported barriers observed by HCPs included difficulty downloading and registering the DMHA; lack of patient motivation, self-direction, and consistent use; time constraints for both patient and HCPs; and follow-up about DMHA use and effectiveness.

Typical Patients for Whom DMHAs Are Referred

Using open-ended text responses, HCPs described “typical patients” commonly referred to DMHAs based on perceived need, technical capability, and common medical conditions. As summarized, perceived need factors included (1) healthy adults who are not yet ready for psychiatry or psychotherapy but would benefit from self-awareness and coaching on their own time; (2) patients looking for ways to understand their own reactions better or wanting to gain insight into their own thought processes as a means for improving anxiety, stress, or depression; and (3) patients needing help with learning cognitive behavioral therapy skills and grounding exercises to help regulate emotion. Perceived technical capability included (1) patients who are smartphone or technology savvy, who already incorporate technology and apps into their lifestyle (eg, Fitbit and music apps), and who use them as part of their entertainment; (2) adolescents and younger patients who are more likely to use DMHAs; and (3) patients who have time to actually use the DMHA and are open to try other resources. In their own words, the top 10 most common medical conditions for which HCPs referred DMHAs included the following: (1) stress, anxiety (mild to severe), and generalized anxiety disorder; (2) depression and major depressive disorder; (3) emotional dysregulation; (4) distorted and negative thinking; (5) posttraumatic stress disorder; (6) sleep disturbance and insomnia; (7) multiple somatic symptoms; (8) traumas from the past; (9) unhappiness; and (10) chronic pain.

Discussion

Overview

This study describes concurrent patient and HCP perspectives within an integrated health system that may affect referral, adoption, and use of DMHAs for improving emotional well-being and mental health conditions in the same practice environment. Importantly, the results reinforce the necessity to identify and tailor individual patient needs and preferences with the most appropriate DMHA to ensure successful engagement and sustainable use. These themes are supported by prior research [24] that identified 16 constructs that may influence user engagement with DMHAs; these constructs are grouped by user (ie, demographics, personal traits, mental health status, beliefs, mental health technology experience, and integration into life), program (ie, type of content, perceived fit, perceived usefulness, level of guidance, social connectedness, and impact of intervention), and technology environment (ie, technology-related factors, privacy and confidentiality, social influence, and implementation) [24]. Many of these constructs manifest themselves in this research and are highlighted in the subsequent sections.

Facilitating Patient Uptake and Sustained Use

Both low and sustained engagement with DMHAs have been recognized as a barrier to successful use [24,25]. In this study, approximately three-fourths (44/58, 76%) of the patients recalled or reported being referred to at least 1 DMHA, suggesting that approximately one-fourth (14/58, 24%) of the patients referred to DMHAs did not engage with them despite documented referral. Although a majority of patients (39/44, 89%) who recalled being referred to a DMHA reported at least some use, only a minority (12/44, 27%) reported >6 months of continued use. It is likely that the time between referral and the survey may have reduced the reported duration of use, making this a conservative estimate, and despite this limitation, the levels of engagement, ever use, and sustained use appear to be higher than those reported in previous research [26,27]. Although only 36% (16/44) of the patient respondents reported current DMHA use, 75% (12/16) of those who did also reported once daily to once weekly use. This finding reinforces the notion that while DMHAs are not a viable or ideal therapeutic solution for everyone, they may have an important role in a select group of patients who routinely use them. Moreover, the ideal or preferred duration of use will likely depend on the condition for which the DMHA is being used. Given the wide array of conditions for DMHA use reported in this pilot research, it is not possible to determine optimal duration for each, and this remains an important area for future research.

Collectively, these findings suggest that initial engagement and sustained use of DMHAs was modest even when an organization encourages and supports their use. If patients are to realize the full benefit of DMHA use, health system interventions that promote and increase identification of patients most likely to engage and sustain DMHA use should be a focus of future efforts. Importantly, 82% (36/44) of the patient respondents reported some degree of helpfulness from the DMHA and >90% (41/44) of the patient respondents agreed or strongly agreed with assuming personal responsibility for their own mental and emotional health and well-being. This aligns with prior research suggesting that DMHAs could be empowering and supportive of self-management [28].

It is recognized that guidance and training facilitate DMHA engagement [24]. On the basis of their own experience, HCP respondents offered practical advice to facilitate successful initiation and sustainable use of DMHAs. It is critical that HCPs invest time, not only at initiation of referral but also during ongoing care, to raise awareness and reinforce why and how DMHA use is an important component of their treatment plan and to encourage engagement. Simply referring a patient to a DMHA with limited explanation is likely insufficient.

It has been previously reported that HCPs who use DMHAs themselves are more likely to use them in practice [29]. In this study, all HCP respondents (12/12, 100%) reported use of a mindfulness and meditation DMHA, but less than half (5/12, 42%) reported use of a cognitive behavioral therapy DMHA. To facilitate use, HCPs recommended using the DMHAs themselves; describing their own experience with the DMHAs; and providing a subsequent show-and-tell with the patient, if comfortable. Assisting the patient with downloading the DMHAs, walking them through how to use it, demonstrating the different features and functionality, and showing them how to find clinically relevant content was also helpful. Collectively, these actions may overcome technology literacy barriers. While these best practices are ideal, time, resource, and reimbursement limitations may foil their implementation. HCPs may become fatigued if they have to perform this for all their patients, which could lead them to stop integrating DMHAs into care if the process is too burdensome. Therefore, alternative solutions and resources to facilitate implementation must be created for sustainability purposes.

Assessing patient interest and highlighting an aspect of the DMHA that aligns with the patient’s area of interest, describing the purpose and value of the DMHA, and ensuring the patient understands that it is an adjunct to in-person therapy and an opportunity to reinforce skills learned in their in-person session may also be beneficial. To support goal setting, HCPs suggested integrating the DMHA into the treatment plan by explaining rationale (eg, mindfulness to decrease anxiety) and indicating that it will be discussed at the next session. Finally, HCP respondents recommended in-person practice using the DMHA and setting reminders for the patient to use the DMHA between sessions, while reassuring ease of use and reinforcing that it does not need to be used daily to further facilitate engagement.

Aligning Patient Preferences and Needs

While patients and HCPs often prioritize the importance of many key DMHA attributes similarly, it is important for HCPs to recognize that a “one-size-fits-all” approach is not ideal [24]. HCPs must identify unique patient preferences rather than using preferences based on their own user experience. In the engagement domain, patients placed highest importance on DMHAs that are fun or interesting to use, whereas HCPs placed less importance on this attribute. Consistent with past reports [24], HCPs placed highest importance on personalized settings and content in the engagement domain, with patients reporting these attributes with second highest importance. With respect to the design domain, patients placed highest importance on visual appeal, while HCPs reported the arrangement and size of buttons, icons, and menus to be most important. The fact that both patients and HCPs in this study reported past or current experience with DMHA use underscores the credibility of these findings related to preferences.

Importantly, previous research recognized that engagement was facilitated with an “appropriate” amount of credible content [24]. In this study, both patients and HCPs reinforced the importance of this attribute by placing it as the highest priority in the information domain. Both patients and HCPs assigned the highest importance to the ease-of-use attribute in the functionality domain. Interestingly, both patients and HCPs assigned the least importance to the DMHA attribute that allows them to be contacted by a mental health HCP, if needed, suggesting a desire for privacy. This observation is supported by the literature that DMHA engagement was improved when there was reassurance that the DMHA provided privacy and anonymity [24].

Finally, patients and HCPs expressed a wide array of patient needs for which the DMHA was referred. Given that it is uncommon for a DMHA to excel in every clinical domain, 42% (5/12) of the HCPs reported that they may order >1 DMHA at an encounter, and 27% (12/44) of the patients indicated that >1 DMHA was recommended by a HCP. These findings underscore the importance of understanding patient needs and preferences and then tailoring the referral of a DMHA to avoid overwhelming the patient and creating potential confusion.

Considering the varying preferences, HCPs must identify the attributes and content that are important and necessary to align the DMHA with each patient’s needs. With a wide array of DMHAs available, HCPs and health systems must develop a decision support system that facilitates alignment of patient clinical needs and their preferred attributes with the best-suited DMHA.

Limitations

While this research is unique in incorporating both patient and HCP perspectives of active DMHA users in a real-world setting, it does have limitations. The small sample, low response rate, and focus within an integrated health system limit generalizability to other practice settings. This research was designed to minimize respondent identification and, therefore, relied on participant self-reported responses, with no linkage to medical records. As such, some variables incurred a number of missing observations (eg, age), but for the most part, missingness was limited. Importantly, the patient sample reflected a slightly older age (42 vs 39 y), similar distributions of White and Black patients (17/44, 39% and 18/44, 41% vs 3473/9525, 36.5% and 3702/9525, 38.9%, respectively), fewer Hispanic patients (2/44, 5% vs 1028/9525, 10.8%), and similar distribution of female patients (32/44, 73% vs 6754/9525, 70.9%) compared to the overall referral pool of 9525 eligible patients during the observation period between April 2021 and December 2021. The HCP sample included a smaller percentage of physicians (7/12, 58% vs 279/410, 68%) and increased percentage of behavioral health care HCPs (6/12, 50% vs 168/410, 41%) than the total pool of 410 eligible HCPs who had referred at least 1 DMHA during a health care encounter between the implementation of the DMHA initiative (December 2019) and December 2021. Despite recruitment, there were no specialty physicians included in the HCP sample. Moreover, the samples were limited to respondents with English language proficiency. In the absence of an ideal survey instrument for purposes of this research, many items were developed de novo but were constructed using acceptable assessment frameworks, evaluated for content validity, and piloted within the study team before use. Given the voluntary nature of this research, a completeness check was not required. However, response consistency was assessed and confirmed by comparing PHQ-4 [21] and Subjective Happiness Scores [22]. Future work should include both HCP and patient end user feedback in instrument design and data collection. Finally, this research mainly focuses on implementation and does not evaluate clinical impact that should be incorporated and studied in future work. Importantly, this research provides a start to a more comprehensive effort to inform clinical decision support and workflow design initiatives that hope to optimize DMHA referral, engagement, sustained use, and clinical effectiveness.

Conclusions

To effectively implement DMHA engagement and sustained use in real-world practice, health systems must develop support systems to identify, align, and tailor individual patient needs and preferences with the most appropriate DMHA. Future research should focus on design of a HCP decision guide that addresses clinician and patient awareness, knowledge, and common misperceptions regarding DMHAs; optimizes alignment of patient need, preferences, and DMHA attributes; encourages engagement and fidelity to DMHA use for widespread adoption across health systems; and subsequently evaluates the impact on mental health outcomes.