This is an ancillary analysis of a pilot clinical trial that examined the feasibility of delivering a digital mental health intervention within the setting of an outpatient clinic visit for the management of chronic pain. As such, participant recruitment and eligibility criteria have previously been described in the pilot study [16].

In brief, participants were recruited between December 2020 and July 2021 from a US tertiary care orthopedic clinic that specializes in the management of chronic musculoskeletal pain. To be eligible, patients had to be 18 years or older and screen positively for symptoms of depression or anxiety by scoring 55 or greater on the Patient-Reported Outcomes Measurement Information System measures for depression or anxiety. Patients who were actively planning to start in-person mental health treatment and those without access to a mobile device were excluded from the study.

Participants were recruited in person by a study coordinator and were offered 8 weeks of complimentary access to the digital intervention (Wysa for Chronic Pain). After providing informed consent, they were given a unique code to access Wysa for Chronic Pain on any Android or iOS phone. Once the code was redeemed, an in-app consent screen appeared before app onboarding began. Throughout the study, no email or phone reminders were sent by the study team outside the app environment for the purposes of app retention or engagement.

The pilot feasibility study received approval from the Washington University institutional review board prior to participant recruitment (202005219), and the study was registered on ClinicalTrials.gov (NCT04640090).

The Wysa for Chronic Pain digital health app is a novel intervention that was specifically developed for people with mental health concerns and coexisting chronic pain. It uses an AI-based digital conversational agent that acts as a supportive companion for users, and it also includes human “coaches” with master’s degrees in counseling, who support users in identifying appropriate therapeutic interventions.

Wysa for Chronic Pain uses CBT principles in the app design and within digital and human conversations. A recurring and key feature of the app is morning and evening conversations (“check-ins”) with the CA (Figure 1), which occur at times chosen by the user. These conversations use behavioral activation principles.

During the morning check-ins, users report their mood on a visual analog scale and commit to engaging in an activity that day that brings them joy. Based on users’ responses, the CA suggests interventions within the app to facilitate pain acceptance and cognitive restructuring. The evening check-ins consist of mood monitoring, and a check on the completion of the planned activity. Based on users’ responses, the CA suggests additional interventions as appropriate and completes the check-in with offering users a bedtime meditation to assist with falling asleep.

The app’s engagement framework primarily rested upon notifications. The timing for the notifications would be chosen by the users, and the app would send these twice a day for the morning and evening check-ins. These would be displayed in the phone notification window, and would bring the user back to the app. Once within the app, the check-in would begin the designed structured intervention. Each week’s efforts would also be encouraged with a weekly report that offered insights and a new tool pack and further guided with a visual roadmap that would outline the path ahead.

It was hypothesized that a consistent and reliable structure that offered scheduled rewards would lead to greater app engagement and retention, along with efficacy.

All engagement and retention data were collected automatically via the app’s usage log. To maintain user anonymity and be consistent with Wysa for Chronic Pain’s confidentiality protocols, demographic data and personally identifiable information were not collected within the app. App sessions were only counted toward engagement and retention if a user completed a conversation with the CA, conversation with the human coach, or an intervention exercise within the app. Passive events, such as opening the app or an element within the app, were not counted.

Engagement metrics included the number of sessions a user completed within (1) each week and (2) during the entire 8-week study period. Retention metrics included the following: (1) retention duration (i.e., users’ last day of engagement with the app during the 8-week study period) and (2) weekly app usage (i.e., interaction with the app at least once each week).

Continuous data were summarized as mean (SD) values and categorical data as n (%) values.

Previous studies have shown that quantifying user retention of an app using survival analysis is feasible. For survival analysis in this trial, the event of interest was defined as complete disengagement (i.e., the day after which a user did not return to the app within the study period). Users whose last day of engagement was after the last day of the study period were right-censored. To minimize blinding to the extent of full engagement in the study period, the Spearman and Kendall rank correlation coefficients were calculated between the total days of engagement and the last day of retention.

Kaplan-Meier nonparametric estimators for computing retention over time and the Cox proportional hazards model were used to understand the impact of morning and evening check-in notifications on user retention over the study period. All analyses included only the prespecified 8-week study period even though users could continue engaging with the app after the conclusion of the study. Data cleaning was performed using Python (version 3.6; Python Software Foundation), and all statistical analyses were performed using R (version 4.1.2; The R Foundation).

Users engaged in a mean of 4.0 (SD 0.9) sessions per week and a mean of 33.3 (SD 42) total sessions during the 8-week study period. When categorized by the type of intervention each tool delivers, the most frequently used interventions were the following: Thought Recording (19.7%), Pain Acceptance (16%), and Sleep Meditations (14.9%) (Table 1).

The median user (N=49) retention period (i.e., time to complete disengagement) was 51 days (95% CI 33-53 days) (Figure 2). The retention rate at 30 days (1 month) was 70%, and 20% of users were right-censored, indicating they were still engaging with the app after completion of the prespecified 8-week study period. A strong correlation was found between users’ total number of days engaged with the app and their last day of retention (Spearman ρ=0.82, P<.001; Kendall τ=0.68, P<.001). Usage of the morning check-in was associated with a longer retention period (hazard ratio=0.89, P=.001) (Table 2).

This study aimed to examine levels of app retention and engagement with Wysa for Chronic Pain, when used by participants managing chronic pain. The app evaluated was an AI-led mental health app (Wysa for Chronic Pain), which used proven in-person health care techniques to improve app engagement. The pilot study that used this intervention has also examined the app’s clinical efficacy while sustaining this engagement [16]. In previous studies on digital health interventions, most participants disengaged within the first week of the study, with average retention periods between 4 and 16 days [17,18]. In comparison, over 50% of users in this study continued returning to the app each week, and the median user retention period was 51 days.

These high rates of engagement and retention suggest that the participants found the app features and interventions helpful and meaningful to their everyday challenges [8]. There is a large body of research examining predictors of user engagement with digital health interventions. The literature indicates that the inclusion of behavior change techniques (e.g., self-monitoring and goal setting) [19,20] and coping strategies (e.g., mindfulness and meditation) [12] is important for the uptake and engagement with digital health interventions. Consistent with this work, our intervention included behavioral activation principles and meditation to support users. Another review based on experiential and behavioral perspectives identified content (e.g., behavior change techniques, social support, and reminders) and content delivery (e.g., professional support, personalization, and aesthetic features) as key factors that may affect engagement with digital health interventions [9]. In line with these findings, our intervention included morning and evening check-ins and personalization of content. Users received customizable notification-type reminders—a feature that has been identified as useful for engagement with digital health interventions [21,22].

Additionally, a recent study found that relational factors predict user engagement with mobile health interventions [23]. The CA in our study used machine learning and AI methods to simulate human-like behaviors and support [10,24]. Considering this, users may have developed a relational connection or a therapeutic alliance with the CA. For example, a recent study demonstrated that Wysa for Chronic Pain users’ therapeutic alliance scores were comparable to ratings from previous studies on alliance in human-delivered face-to-face psychotherapy with clinical populations [25].

Our achievement of high user retention and engagement with a CA-based, pain-customized, digital mental health intervention has clear implications for addressing issues of suboptimal engagement with digital health interventions and of improving access to care for chronic pain. Research indicates that people with chronic pain are almost twice as likely to drop out from self-guided digital interventions when compared to traditional, guided interventions [21]. Despite this, the present trial achieved high levels of retention and engagement among a sample of individuals with chronic pain. Based on the literature highlighting practical strategies and recommendations to tackle issues related to user attrition in digital health interventions, we incorporated behavior change techniques, coping strategies, and morning and evening check-ins. From our results, it is clear this combination of engagement strategies and therapeutic content has the potential to retain and engage users, which may be related to the enhanced effectiveness of the intervention. Future work should use these findings to inform the design of evidence-based interventions for individuals with chronic pain and to enhance user retention and engagement of digital health interventions more broadly.

The key limitations of this study are its small sample size, the lack of demographic analysis (which reduces information about the retained users), and the limited generalizability. The event of interest within the survival analysis (i.e., defined as complete disengagement) partially blinded the data to the number of breaks until the last day of engagement. With a larger sample size, a more granular analysis could evaluate each streak within the data set, leading up to the last day of engagement. Additionally, there was no comparison group, which reduced the ability to establish causality for the possible factors responsible for engagement and retention.

Given the promising results, future research should be undertaken with a larger sample size and comparative groups. It should also be determined if these results can be replicated with populations managing other health conditions and for users who were not initially introduced to the app through a medical clinic setting.

This study highlights the ability to attain high retention and engagement with a mental health app that uses a digital CA and established behavioral paradigms that improve engagement with in-person therapy, specifically to deliver CBT to people with mental health concerns and coexisting chronic pain.