This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Digital applications are commonly used to support mental health and well-being. However, successfully retaining and engaging users to complete digital interventions is challenging, and comorbidities such as chronic pain further reduce user engagement. Digital conversational agents (CAs) may improve user engagement by applying engagement principles that have been implemented within in-person care settings.
To evaluate user retention and engagement with an artificial intelligence–led digital mental health app (Wysa for Chronic Pain) that is customized for individuals managing mental health symptoms and coexisting chronic pain.
In this ancillary survival analysis of a clinical trial, participants included 51 adults who presented to a tertiary care center for chronic musculoskeletal pain, who endorsed coexisting symptoms of depression or anxiety (Patient-Reported Outcomes Measurement Information System score of ≥55 for depression or anxiety), and initiated onboarding to an 8-week subscription of Wysa for Chronic Pain. The study outcomes were user retention, defined as revisiting the app each week and on the last day of engagement, and user engagement, defined by the number of sessions the user completed.
Users engaged in a cumulative mean of 33.3 sessions during the 8-week study period. The survival analysis depicted a median user retention period (i.e., time to complete disengagement) of 51 days, with the usage of a morning check-in feature having a significant relationship with a longer retention period (
Our findings suggest that user retention and engagement with a CBT-based CA built for users with chronic pain is higher than standard industry metrics. These results have clear implications for addressing issues of suboptimal engagement of digital health interventions and improving access to care for chronic pain. Future work should use these findings to inform the design of evidence-based interventions for individuals with chronic pain and to enhance user retention and engagement of digital health interventions more broadly.
ClinicalTrials.gov NCT04640090; https://clinicaltrials.gov/ct2/show/NCT04640090
Digital health interventions offer an opportunity to reduce health care barriers for individuals with chronic pain by increasing accessibility and decreasing cost- and time-related barriers to care [
Engagement refers to the extent of usage and the perceived desire to use the digital intervention over a prolonged time period [
Key factors that support engagement and retention within in-person clinical care settings include patient-clinician agreement, patients’ perception of listening and empathy by clinicians, collaborative learning mechanisms for patients, and reminder systems [
The purpose of this study was to examine user retention and engagement with Wysa for Chronic Pain—a digital CA based on CBT principles—that is designed to improve general well-being among people with chronic pain.
This is an ancillary analysis of a pilot clinical trial that examined the feasibility of delivering a digital mental health intervention within the setting of an outpatient clinic visit for the management of chronic pain. As such, participant recruitment and eligibility criteria have previously been described in the pilot study [
In brief, participants were recruited between December 2020 and July 2021 from a US tertiary care orthopedic clinic that specializes in the management of chronic musculoskeletal pain. To be eligible, patients had to be 18 years or older and screen positively for symptoms of depression or anxiety by scoring 55 or greater on the Patient-Reported Outcomes Measurement Information System measures for depression or anxiety. Patients who were actively planning to start in-person mental health treatment and those without access to a mobile device were excluded from the study.
Participants were recruited in person by a study coordinator and were offered 8 weeks of complimentary access to the digital intervention (Wysa for Chronic Pain). After providing informed consent, they were given a unique code to access Wysa for Chronic Pain on any Android or iOS phone. Once the code was redeemed, an in-app consent screen appeared before app onboarding began. Throughout the study, no email or phone reminders were sent by the study team outside the app environment for the purposes of app retention or engagement.
The pilot feasibility study received approval from the Washington University institutional review board prior to participant recruitment (202005219), and the study was registered on ClinicalTrials.gov (NCT04640090).
The Wysa for Chronic Pain digital health app is a novel intervention that was specifically developed for people with mental health concerns and coexisting chronic pain. It uses an AI-based digital conversational agent that acts as a supportive companion for users, and it also includes human “coaches” with master’s degrees in counseling, who support users in identifying appropriate therapeutic interventions.
Wysa for Chronic Pain uses CBT principles in the app design and within digital and human conversations. A recurring and key feature of the app is morning and evening conversations (“check-ins”) with the CA (
During the morning check-ins, users report their mood on a visual analog scale and commit to engaging in an activity that day that brings them joy. Based on users’ responses, the CA suggests interventions within the app to facilitate pain acceptance and cognitive restructuring. The evening check-ins consist of mood monitoring, and a check on the completion of the planned activity. Based on users’ responses, the CA suggests additional interventions as appropriate and completes the check-in with offering users a bedtime meditation to assist with falling asleep.
The app’s engagement framework primarily rested upon notifications. The timing for the notifications would be chosen by the users, and the app would send these twice a day for the morning and evening check-ins. These would be displayed in the phone notification window, and would bring the user back to the app. Once within the app, the check-in would begin the designed structured intervention. Each week’s efforts would also be encouraged with a weekly report that offered insights and a new tool pack and further guided with a visual roadmap that would outline the path ahead.
It was hypothesized that a consistent and reliable structure that offered scheduled rewards would lead to greater app engagement and retention, along with efficacy.
The conversational agent asks the user for a check-in time.
All engagement and retention data were collected automatically via the app’s usage log. To maintain user anonymity and be consistent with Wysa for Chronic Pain’s confidentiality protocols, demographic data and personally identifiable information were not collected within the app. App sessions were only counted toward engagement and retention if a user completed a conversation with the CA, conversation with the human coach, or an intervention exercise within the app. Passive events, such as opening the app or an element within the app, were not counted.
Engagement metrics included the number of sessions a user completed within (1) each week and (2) during the entire 8-week study period. Retention metrics included the following: (1) retention duration (i.e., users’ last day of engagement with the app during the 8-week study period) and (2) weekly app usage (i.e., interaction with the app at least once each week).
Continuous data were summarized as mean (SD) values and categorical data as n (%) values.
Previous studies have shown that quantifying user retention of an app using survival analysis is feasible. For survival analysis in this trial, the event of interest was defined as complete disengagement (i.e., the day after which a user did not return to the app within the study period). Users whose last day of engagement was after the last day of the study period were right-censored. To minimize blinding to the extent of full engagement in the study period, the Spearman and Kendall rank correlation coefficients were calculated between the total days of engagement and the last day of retention.
Kaplan-Meier nonparametric estimators for computing retention over time and the Cox proportional hazards model were used to understand the impact of morning and evening check-in notifications on user retention over the study period. All analyses included only the prespecified 8-week study period even though users could continue engaging with the app after the conclusion of the study. Data cleaning was performed using Python (version 3.6; Python Software Foundation), and all statistical analyses were performed using R (version 4.1.2; The R Foundation).
Of the 51 study participants who redeemed an app code for the trial, 49 (96%) onboarded successfully to the app.
Users engaged in a mean of 4.0 (SD 0.9) sessions per week and a mean of 33.3 (SD 42) total sessions during the 8-week study period. When categorized by the type of intervention each tool delivers, the most frequently used interventions were the following: Thought Recording (19.7%), Pain Acceptance (16%), and Sleep Meditations (14.9%) (
The most frequently used interventions in Wysa for Chronic Pain.
Intervention | Usage, % |
Thought Recording | 19.7 |
Pain Acceptance | 16.0 |
Sleep Meditation | 14.9 |
Mindfulness | 10.4 |
Anxiety Management | 9.8 |
Gratitude | 8.1 |
Motivational Stories | 7.3 |
Calming Exercises | 6.5 |
Social Support | 2.0 |
Grounding | 1.4 |
The median user (N=49) retention period (i.e., time to complete disengagement) was 51 days (95% CI 33-53 days) (
Kaplan-Meier survival curve modeling user retention in the Wysa for Chronic Pain app during the 8-week study period. Dashed lines represent 95% CIs of the survival curve. Model concordance=0.829 (SE 0.045,
Cox proportional hazards model to explore the impact of engaging with morning and evening check-ins on user retention.
Variable | Hazard ratio | SE | |
Morning check-in | 0.894 | 0.035 | .001a |
Evening check-in | 0.987 | 0.024 | .59 |
aStatistically significant.
This study aimed to examine levels of app retention and engagement with Wysa for Chronic Pain, when used by participants managing chronic pain. The app evaluated was an AI-led mental health app (Wysa for Chronic Pain), which used proven in-person health care techniques to improve app engagement. The pilot study that used this intervention has also examined the app’s clinical efficacy while sustaining this engagement [
These high rates of engagement and retention suggest that the participants found the app features and interventions helpful and meaningful to their everyday challenges [
Additionally, a recent study found that relational factors predict user engagement with mobile health interventions [
Our achievement of high user retention and engagement with a CA-based, pain-customized, digital mental health intervention has clear implications for addressing issues of suboptimal engagement with digital health interventions and of improving access to care for chronic pain. Research indicates that people with chronic pain are almost twice as likely to drop out from self-guided digital interventions when compared to traditional, guided interventions [
The key limitations of this study are its small sample size, the lack of demographic analysis (which reduces information about the retained users), and the limited generalizability. The event of interest within the survival analysis (i.e., defined as complete disengagement) partially blinded the data to the number of breaks until the last day of engagement. With a larger sample size, a more granular analysis could evaluate each streak within the data set, leading up to the last day of engagement. Additionally, there was no comparison group, which reduced the ability to establish causality for the possible factors responsible for engagement and retention.
Given the promising results, future research should be undertaken with a larger sample size and comparative groups. It should also be determined if these results can be replicated with populations managing other health conditions and for users who were not initially introduced to the app through a medical clinic setting.
This study highlights the ability to attain high retention and engagement with a mental health app that uses a digital CA and established behavioral paradigms that improve engagement with in-person therapy, specifically to deliver CBT to people with mental health concerns and coexisting chronic pain.
artificial intelligence
conversational agent
cognitive behavior therapy
CS conceptualized the ancillary analysis and contributed to the writing of the manuscript. ALC reviewed the data and contributed to the final manuscript. MK contributed to the statistical methods and analysis of the study.
CS and MK are reported as being employees of Wysa Inc and owning equity in the company. ALC declares no conflicts of interest related to the study and is funded by the National Institutes of Health and the National Institute of Arthritis and Musculoskeletal and Skin Diseases (K23AR074520) and the Doris Duke Charitable Foundation. The pilot clinical trial was funded by a Washington University/BJC HealthCare Big Ideas innovation grant. Of note, the pilot clinical trial was conducted by ALC, and prespecified results were reported independent of input from Wysa Inc. This manuscript describes an ancillary analysis of data that were collected during the conduct of the pilot clinical trial.