Poor mental health is increasing, affecting 42.5% of US adults, according to a 2022 survey [1]. People face barriers to accessing traditional care, such as a lack of service availability and high costs [2,3]. Digital mental health interventions (DMHIs) have been shown to improve mental health [4-7], with evidence highlighting their potential to help users achieve their mental health goals (eg, symptom reduction or the development of new coping skills) [8,9]. Engagement is considered a key mediator of outcomes [9-12]. However, DMHIs currently experience low engagement and high dropout rates [7,13-15]. Among commercial DMHIs, the median retention rate was 3.9% after 15 days of use [16], indicating a need to both understand and improve user engagement with these tools [14,17]. The challenge of low engagement with DMHIs is further complicated by a lack of consensus on what engagement means and a lack of understanding of how engagement is applied in their design [10,17,18].

Engagement with digital interventions is a multidimensional concept often discussed as being composed of 3 constructs: cognitive (eg, effort and attention), affective (eg, emotions and values), and behavioral (eg, usage and actions) engagement [2,4,19-21]. However, no standard definition of engagement exists across research fields, such as behavioral sciences and human-computer interaction (HCI) [10,18,22]. Existing models of engagement highlight the dynamic process of user engagement with technology that goes beyond discrete technology use [10,23-25]. The foundational model of engagement by O’Brien and Toms [24] presents the temporal stages of engagement, starting with the point of engagement, sustained engagement, end of engagement, and, sometimes, re-engagement. Each stage is characterized by attributes of user experience (eg, positive affect and sensory appeal), based on interviews with users of video games, educational applications, online shopping, and web search [24]. As an extension of this dynamic view, Yardley et al [26] differentiated between the micro- and macro-level engagement. Microengagement, also known as “little e,” refers to discrete or momentary engagement with the digital intervention, particularly with the technology [26,27], including the user interface elements and the behavior change intervention components that are presented in the technology [27]. Macroengagement refers to the user’s engagement with the behavior change process that the DMHI is designed to support [26,27]. Together, these levels capture both discrete and longitudinal contexts of how users engage with digital systems. Our study examined how the micro and macro levels of engagement are applied in the design of DMHIs.

Research on engagement has mainly focused on conceptualizing and measuring engagement rather than on understanding how the definitions are reflected in the practice of creating technologies [10,17,18,23,24]. There are engagement models specific to DMHIs, but they tend to focus on measuring or predicting engagement outcomes of a single intervention [28,29]. For example, the Integrative Engagement Model of Digital Psychotherapy [29] combines frameworks from behavioral sciences and HCI to define the phases of engagement with therapist-supported DMHIs. These models advance the understanding of engagement with DMHIs, but do not inform how researchers operationalize engagement. Lipschitz et al [17] emphasized that research needs to advance theories of what drives DMHIs engagement and that this needs to be understood at different levels, in real-world settings, rather than only in clinical trials.

How engagement is measured reflects these conceptual tensions. There is no standard measurement approach for engagement, but quantitative measures are the most commonly used [17,30-32]. Quantitative measures are often used to measure engagement at discrete moments in time [10,25,33] through methods such as log analysis (eg, number of clicks and frequency of views) [10,34]. However, these metrics predominantly capture usage and fail to account for the dynamic process that models describe. There has been an ongoing discussion about expanding beyond traditional quantitative usage metrics, with more mixed methods approaches to measuring engagement (eg, qualitative interviews [10]), to help surface the nuanced and dynamic aspects of engagement [10,23-25].

The challenges of conceptualizing and measuring engagement are amplified by the nature of mental health. Mental health symptoms are dynamic, fluctuating over time [35], and can affect user engagement due to a lack of intrinsic motivation [36] and life barriers [36,37]. Incorporating human support or a human-in-the-loop approach for technical support, accountability, or encouragement is a common strategy to sustain engagement with DMHIs [10,38-40]. Although these strategies tend to enhance engagement, DMHI engagement still remains low, with high dropout rates [7,13,15,41].

This research focuses on defining engagement constructs and outcome-focused models, but it has not investigated how professionals integrate engagement into the design of DMHIs. Without understanding how engagement is put into practice, even a tested framework might not succeed in improving engagement due to inconsistent applications of engagement measures and strategies [17]. This study aims to expand our understanding of how academic clinical researchers operationalize engagement in DMHIs. To achieve this, we interviewed clinical mental health researchers who created DMHIs for their research projects (ie, pilots and clinical trials). As a first step in understanding how engagement is operationalized in DMHI development, we interviewed clinical researchers because they often lead the development of DMHIs and make decisions about how engagement will be operationalized for these systems. Since little is known about how engagement is operationalized in this context, we used a qualitative interviewing method, as it allowed us to examine in-depth how clinical researchers operationalize engagement. We viewed this approach as particularly useful because the rationales and processes underlying these operationalization decisions are often implicit, nuanced, and context-dependent. This study aims to better understand how clinical researchers operationalize engagement for DMHIs and to discuss strategies to better align these operationalizations with the dynamic nature of user engagement with DMHIs.

We conducted semistructured interviews with 12 clinical mental health researchers based in the United States who develop and evaluate DMHIs. All DMHI projects were part of academic research projects, either as research pilots or as clinical trials. Participants were recruited through a purposive sampling of researchers within the professional networks of the authors. Additional participants were recruited through a snowball strategy from the initial group [42]. We contacted all participants by email, and all researchers contacted agreed to participate. Participants were eligible for the study if they (1) were mental health experts, (2) worked in academia, (3) had directly worked on or supervised digital mental health intervention projects, (4) were based in the United States, (5) spoke English, and (6) were over 18 years of age.

This study was registered and classified as exempt by the Institutional Review Board of the University of California, Irvine (protocol 2761). All participants were provided with written informed consent via email prior to participating in the study and were invited via email to schedule an interview if they agreed to participate. Participants were informed that they could withdraw from the study at any time without penalty. Participants were compensated with a US $30 gift card after the interview. All data collected were deidentified before any analysis, and only the researchers assigned to this study had access to the data.

We started the semistructured interviews with participants after iterations of the interview protocol. We conducted 1 pilot interview. BO and KN conducted semistructured interviews via Zoom (Zoom Communications, Inc) that lasted 45 to 60 minutes. The interviews were conducted between May 2023 and February 2024. The authors, BO and KN, along with the participants, were present during the interviews. BO and KN were more junior than all participants (ie, a PhD candidate and a senior undergraduate student, respectively). BO and KN introduced themselves at the start of each interview, with one author conducting the interview, while the other turned off their camera and took notes. If the note-taking author had follow-up questions, they asked them at the end of the interview. All 12 interviews were video and audio-recorded with participants’ consent, transcribed, and analyzed. We transcribed all interviews using Otter.ai (Otter.ai, Inc), and the 2 interviewers reviewed and edited the transcripts for accuracy by comparing them to the recordings. The transcripts were not returned to participants for comment. Recruitment, data collection, and analysis were conducted concurrently until we reached saturation on the themes discussed in this study [43]. For instance, themes such as “engagement conceptualized as amount of usage” recurred frequently and were defined early in the interview process, with limited new insights emerging in later interviews. Other themes, such as “engagement that is not captured” (ie, invisible engagement) and “challenges to finding opportune moments for engagement,” appeared less frequently in our analysis but consistently recurred across participants with sufficient depth and consistency to support the development of these themes. By the end of the 12 interviews, we were no longer generating substantial new insights related to the study objectives.

We conducted a thematic analysis within a constructivist framework [44]. Thematic analysis involves the process of researchers familiarizing themselves with the data, generating initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the report [44]. Two researchers used inductive coding to analyze the interviews [45] using the software ATLAS.ti (ATLAS.ti Scientific Software Development GmbH). First, each researcher familiarized themselves with the data and inductively coded 3 transcripts independently, and then compared interpretations [44]. We frequently iterated on the codes and discussed them with the entire research team as part of the thematic analysis process to make sense of the data and discuss potential themes that go beyond simple descriptions of the content [44,46]. Those codes were used to create a codebook, which included codes such as “experience creating digital tools,” “objective engagement,” and “intervention and technology together.” The 2 researchers continued to review codes weekly with each other and other team members until all transcripts had been reviewed. During the coding process, we started searching for themes [44]. We also took notes after interviews and wrote memos as we read the transcripts to discuss and resolve disagreements about codes and themes. The first iteration of affinity mapping to develop and review potential themes was done in person with physical sticky notes, which slowed the process and allowed the researchers to discuss the organization of codes and patterns [44,47]. Then, BO conducted pattern coding [45]. This process led to the conceptualization of themes such as “measurement of engagement” and “intervention vs technology.” As we produced this manuscript, we iterated on the themes. We did not share the data analysis and themes with participants.

We interviewed 12 participants (5 men and 7 women). All participants were academic faculty. All participants had experience using participatory human-centered design (HCD) approaches and applied those approaches in their DMHI projects. Table 1 includes the background of participants, the focus of their DMHI, the modality of technology used for the DMHI (eg, SMS, app, and website), the development stage (prototype, pilot, or randomized controlled trial), and the target user group. A prototype refers to early versions of the DMHI, often cocreated with users; pilot refers to feasibility or pilot trials, during which the DMHI is refined; and randomized controlled trial refers to relatively advanced versions of the DMHI.

In this section, we first describe how clinical researchers operationalized engagement based on clinical outcomes but have started to include aspects of user experience. Then, we provide an overview of the engagement dimensions and specific findings for each of the dimensions. Finally, we present the barriers clinical researchers face in finding opportune moments for users to engage with DMHIs.

Participants highlighted that the clinical outcome, regardless of the amount or quality of user engagement with the technology, was their primary goal. In their pilots and clinical trials, engagement was not a main consideration in the design or evaluation of DMHIs. Instead, the focus was on the clinical outcomes because they are the traditional clinical research objectives and were also tied to the objectives outlined in their research grants. P5 stated, “Engagement was probably a secondary or tertiary consideration.” P5 explained that when considering engagement for DMHIs, “It was more about interface,” representing a common view among participants that engagement is tied primarily to the technology component of the DMHI.

Successful clinical outcomes meant achieving behavior change and clinically meaningful symptom reduction. Clinical researchers wanted users to sustain engagement with the learned behavior in their daily lives (ie, behavior change):

Participants prioritized clinical outcomes over user engagement with the DMHI. Their goal was for users to achieve their mental health goals rather than to improve user engagement with the DMHI.

As their research projects progressed, participants’ understanding of engagement with DMHIs changed. At first, participants relied on their clinical expertise. Later, as participants worked more with HCI researchers and/or incorporated HCD methods (eg, co-design workshops and feedback sessions), they recognized the importance of the user experience in enhancing engagement with DMHIs. A participant reflected on their exclusive reliance on clinical experience and realized they needed to talk to users:

A participant said they had learned to prioritize user experience as well as adhere to psychotherapeutic principles:

Participants said their understanding of engagement evolved to recognize the preferences and experiences of users:

As participants gained more experience in HCI and HCD approaches, their understanding of engagement evolved from focusing on clinical aspects, such as adherence, to including the user experience.

We separate engagement, as described by participants, into different dimensions reflecting its complex nature based on our findings (Textbox 1). These dimensions include DMHI components (intervention, technology, and human support) [38,48], levels of engagement (micro and macro) [26,27], and visibility of engagement (visible and invisible).

Participants discussed concepts aligned with the micro- and macroengagement levels proposed by Yardley et al [26] (Textbox 1). However, they did not use this terminology; instead, they discussed users’ moment-to-moment interactions with the DMHI itself or users’ involvement with the intervention processes and goals. Participants measured engagement success in terms of clinically meaningful outcomes, which happen through macroengagement (eg, practicing therapeutic skills in and outside the app). However, many of the measurement tools primarily measured engagement at the micro level (eg, logging into the app). All participants used quantitative metrics to evaluate their DMHI, with some also incorporating qualitative methods. Quantitative measures of engagement traditionally focused on usage data (eg, number of clicks or the amount of usage of different features):

Participants emphasized that clinically meaningful outcomes required more macroengagement (ie, engagement with the intervention goals) and not necessarily frequent or prolonged engagement with the technology. One participant explained:

Given participants’ focus on macroengagement as a way to achieve meaningful clinical outcomes, some participants expressed skepticism about the ability of quantitative metrics alone to fully capture whether users are engaging with the intervention’s goals:

Some participants discussed the need for a shift toward using more qualitative approaches (eg, user satisfaction) to complement the understanding of engagement. To gain deeper insights into engagement, especially macroengagement, a participant said that it is important to understand how users think and feel about the DMHI:

Another perspective on engagement levels was that even a little microengagement could lead to a successful engagement. Some participants considered a successful engagement with DMHIs to be any progress that users made toward their goals (eg, behavior change and symptom reduction), even if the users had not yet achieved their main mental health goal (ie, intervention outcome). A participant stated that even a little engagement with the DMHI (ie, microengagement) could propel someone to get better when they are in their worst mental health state:

That perspective highlights how a quantitative-only approach would be insufficient to understand user engagement with the DMHI because it misses the qualitative aspects (eg, user satisfaction, feelings, and relationship with the DMHI) that lead users toward macroengagement.

Participants said it was challenging to match the delivery time of a DMHI with opportune moments for engagement. For example, a DMHI involved text messages that users needed to reply to so that engagement would happen. In that case, opportune moments were the moments when users were available to reflect on and reply to the text message. The participants noticed that they had better response rates when the time at which the intervention was sent intersected with an opportune moment for the users to answer:

Some participants asked users to select their preferred time for receiving messages, aiming to enhance engagement at opportune moments:

However, asking users for their preferred time to receive reminders did not completely solve the problem of reaching users at the right time. To address this, a participant was exploring ways to reach users at opportune moments for engagement, based on user needs and context, but they had not yet found a good solution:

Participants highlighted that delivering the intervention when users were available and willing to engage improved response rates and engagement. However, understanding each user’s opportune moments for engagement, which are often invisible, remained a major challenge.

Clinical mental health researchers who created DMHIs navigated tensions between the complexity of engagement dimensions, users’ uncaptured experiences, and their project constraints. Participants focused on measuring microengagement activities, but their project goals were tied to macroengagement activities. This issue highlights one of the main challenges in clearly operationalizing engagement for DMHIs—the disconnect between what is being measured and the goals of the DMHI, which reflects the broader lack of consensus on how engagement should be operationalized. In this section, we discuss the challenges of designing for engagement and suggest design considerations based on participants’ research practices and barriers. Our findings unpack engagement into dimensions that can guide the choice of engagement strategies and design decisions for DMHI projects.

DMHI researchers faced a tension between measuring clinical outcomes and understanding user engagement. One of the key differences between clinical outcomes and user engagement is that clinical outcomes are measured by end point results, captured in the controlled environment of a clinical trial, whereas user engagement is a dynamic process of users’ interactions with the DMHI and is best understood in real-world settings. Researchers primarily captured engagement at the micro level, which provided a snapshot of users’ moment-by-moment interactions with the technology. However, this often failed to capture the contextual and longitudinal complexities of engagement [10,26,50] with DMHIs. This issue can be addressed by incorporating a macro-level perspective (ie, users’ engagement with the broader intervention goals) that includes longitudinal user experiences, context, and changes over time [26,27,50,51]. Focusing on macro-level engagement could help in understanding the dynamic nature of mental health in relation to longitudinal engagement with the DMHI.

The tension between the focus on clinical outcomes and users’ experiences leads to different understandings of “successful engagement.” Due to the focus on clinical outcomes, clinical research frameworks often overlook some of the benefits that users gain from DMHIs. In contrast, some participants considered an intervention successfully engaging for users even when it led to small benefits (eg, taking a first step for behavior change), regardless of whether the intervention achieved the intended outcome. This perspective shifts the understanding of “successful engagement” from solely focusing on clinical outcomes to also recognizing small user progress and subjective benefits, which aligns with HCD and HCI literature [18]. Prior research indicates that understanding how users engage with a DMHI (eg, how much control they thought the DMHI gave them over activities in their lives; how easy they thought it was to use the DMHI) can provide better insights into the mechanisms of change of a DMHI rather than just quantitative usage metrics. This is because usage data were found not to be significantly related to clinical outcomes [52]. That highlights the importance of using a mix of quantitative and qualitative data to capture how users engage, the user experience, and the clinical outcomes in both clinical research and real-world settings. However, despite recognizing the value of qualitative approaches, our findings show that participants found it difficult to prioritize those methods in clinical research settings due to the need for more time and resources (eg, researchers and funding).

The dimensions of engagement (Textbox 1) can help researchers better prioritize aspects of engagement that are important to them when designing DMHIs. Researchers can examine individual dimensions (technology, intervention, and human support components; micro- and macro-levels of engagement; and visible and invisible engagement) to determine which one(s) may need to be addressed to help them reach their design goals. For example, at the component level, the technology may need to be adapted for users unfamiliar with digital systems, or an intervention may need to include terminology that is culturally appropriate [53,54]. Similarly, if the goal is for users to apply therapeutic skills in their daily lives (ie, macroengagement), researchers should prioritize understanding users’ contexts and behaviors outside the app rather than focusing only on in-app interactions. At the same time, while focusing on individual dimensions to understand their impact, it is just as important to understand how these dimensions affect one another. Overlooking these interdependencies can make a design decision made for one component unintentionally hinder engagement in another. For example, in an area with poor internet service, the need for a lightweight website (ie, technology) affects the design of the content (ie, intervention). Consequently, we suggest that researchers explicitly map, early on, which dimensions and interdependencies are most important for their DMHI. These decisions will affect resource allocation and the choice of relevant design and measurement approaches.

Participants reported that one of their main challenges was capturing users’ changing behavior and context when engaging with the DMHI. Based on our findings, visible engagement refers to the user engagement that researchers can measure, such as engagement conducted digitally and captured by log data, or that happens in person in a setting (eg, a hospital) where researchers can observe these activities. This includes usage data in an app or interactions with a coach. In contrast, invisible engagement (Table 2) refers to user engagement that is not directly observable and not easily measured, such as when users practice their target behavior outside the DMHI app or intervention environment. However, invisible engagement can become visible if it is captured; for example, if researchers are able to track that a person is working on the intervention activity outside of the app.

Capturing invisible engagement could help identify opportune moments of engagement, thereby increasing user response rates and engagement with the DMHI, which could improve clinical outcomes. Currently, participants mostly rely on users to indicate to them when it would be a good time to receive an intervention activity, but barriers can unexpectedly hinder engagement. By capturing invisible engagement, researchers could understand contextual triggers, such as users’ emotional states, environments, and times that support or hinder engagement, and deliver future interventions at moments when users are more likely to engage.

Therefore, the best approach for researchers aiming to capture invisible engagement with a DMHI is through a combination of methods. Participants used methods such as ecological momentary assessments and questionnaires. However, participants stated that these could lead to inaccurate insights into invisible engagement because they rely on the user’s memory and self-report. Ecological momentary assessments have the benefit of assessing longitudinal experiences and reducing recall bias, but they still rely on self-report and may be burdensome to users [55]. Passive sensing methods to collect information on phone usage patterns, activities, or location can offer researchers a way to capture physiological data and more contextual information without increasing the burden on users. Qualitative approaches could add depth to the understanding of the different types of engagement. For example, microphenomenological interviews, which focus on interviewing and analyzing a specific subjective experience [56], could yield insights into cognitive and affective engagement that can be harder for users to explain without this specific probing. Capturing more information from some users about what is usually invisible gives more researchers greater insights into how to continue iterating on the design to improve and maintain engagement after the DMHI is implemented.

Nonetheless, several aspects of invisible engagement can be difficult to make visible with current approaches. Technical, administrative, and ethical barriers, such as sensors breaking, which results in incomplete or unreliable data, data breaches and privacy concerns [26,57], and limited resources to apply additional methods, limit what is feasible to capture. The goal should be to make selective aspects of engagement visible, specifically the ones that researchers hypothesize to affect outcomes and user experience, rather than trying to capture all aspects of invisible engagement.

Through our data analysis, we have identified 2 different dimensions of engagement. First, as described in the Results section, we found that engagement occurred at both the macro- and micro-levels. Yardley et al [26] have called for researchers to operationalize engagement for digital behavior change interventions at the micro- and macro-levels. We add to this understanding by illustrating how these dimensions are reflected in digital mental health interventions, focusing macro-level engagement on activities related to behavior change (ie, using therapeutic skills during the week without tracking them in the app) and the micro-level engagement on activities related to technology use (ie, clicking on features in the app).

Our analysis highlights that participants viewed microengagement as interactions that are more readily captured and, therefore, visible, whereas macroengagement was more difficult to capture and, therefore, often invisible. However, the relationship between these 2 different dimensions of engagement may not be a simple 1:1 correspondence, as suggested by our participants. Future research should further examine these relationships across different intervention contexts.

Future work should investigate the process of how engagement occurs during specific moments of interaction with a DMHI, including invisible engagement, to inform the design of systems that enhance engagement. This could include studies with potential users of DMHIs, examining the cognitive, affective, and behavioral constructs of engagement within each dimension, as well as the interactions between dimensions.

This study has a few limitations. As a qualitative study, the findings may not be generalizable to all researchers creating and implementing DMHIs. Moreover, all participants were academic researchers in the United States. Academic institutions in the United States face specific academic constraints and operational requirements tied to grant funding and clinical trials, which could affect how engagement is operationalized. Specifically, these constraints can limit the time or financial resources available to prioritize engagement during project design. Therefore, the results may not represent industry settings or cultural contexts outside of the United States. BO and KN conducted the interviews together. The presence of 2 interviewers may have introduced social desirability bias. To partially mitigate that bias, the researcher who was not interviewing turned off their camera and microphone after introducing themselves and took notes until close to the end of the interview. Finally, the interviews were mostly conducted after the engagement aspects of the projects had concluded, so the findings were based on recall. We did not use any mitigation strategies for recall bias, such as reviewing project documents or sending follow-up emails. Consequently, participants may have omitted details or misremembered aspects of their projects during the interview.

This study contributed to the understanding of engagement in DMHIs. Building on prior conceptualization, we proposed that 3 dimensions of engagement (ie, components, levels, and visibility) can guide more intentional design for DMHIs and other health-related technologies. By introducing visibility as a dimension of engagement, our work highlights aspects of engagement that occur beyond what researchers can readily observe and measure. Specifically, this study highlighted a tension between clinical mental health researchers’ goals and the qualitative and experiential aspects of engagement that are often not captured in DMHIs. Researchers created DMHIs with macroengagement as a goal, but mainly captured microengagement through quantitative usage metrics and end-point results, missing the qualitative dynamics of user engagement in real-world settings. Making visibility explicit may support researchers in accounting for aspects of engagement that matter most for their research goals but are harder to capture.