Published on in Vol 6, No 7 (2022): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/36018, first published .
Improving Well-being With a Mobile Artificial Intelligence–Powered Acceptance Commitment Therapy Tool: Pragmatic Retrospective Study

Improving Well-being With a Mobile Artificial Intelligence–Powered Acceptance Commitment Therapy Tool: Pragmatic Retrospective Study

Improving Well-being With a Mobile Artificial Intelligence–Powered Acceptance Commitment Therapy Tool: Pragmatic Retrospective Study

Authors of this article:

Navot Naor1 Author Orcid Image ;   Alex Frenkel1 Author Orcid Image ;   Mirène Winsberg2 Author Orcid Image

Original Paper

1Kai.ai, Tel Aviv, Israel

2Kai.ai, San Francisco, CA, United States

Corresponding Author:

Navot Naor, PhD

Kai.ai

Yigal Alon St 114

Tel Aviv, 6744320

Israel

Phone: 972 502183965

Email: navot@kai.ai


Background: Research and dissemination of smartphone apps to deliver coaching and psychological driven intervention had seen a great surge in recent years. Notably, Acceptance Commitment Therapy (ACT) protocols were shown to be uniquely effective in treating symptoms for both depression and anxiety when delivered through smartphone apps. The aim of this study is to expand on that work and test the suitability of artificial intelligence–driven interventions delivered directly through popular texting apps.

Objective: This study evaluated our hypothesis that using Kai.ai will result in improved well-being.

Methods: We performed a pragmatic retrospective analysis of 2909 users who used Kai.ai on one of the top messaging apps (iMessage, WhatsApp, Discord, Telegram, etc). Users’ well-being levels were tracked using the World Health Organization-Five Well-Being Index throughout the engagement with service. A 1-tailed paired samples t test was used to assess well-being levels before and after usage, and hierarchical linear modeling was used to examine the change in symptoms over time.

Results: The median well-being score at the last measurement was higher (median 52) than that at the start of the intervention (median 40), indicating a significant improvement (W=2682927; P<.001). Furthermore, HLM results showed that the improvement in well-being was linearly related to the number of daily messages a user sent (β=.029; t81.36=4; P<.001), as well as the interaction between the number of messages and unique number of days (β=–.0003; t81.36=–2.2; P=.03).

Conclusions: Mobile-based ACT interventions are effective means to improve individuals’ well-being. Our findings further demonstrate Kai.ai’s great promise in helping individuals improve and maintain high levels of well-being and thus improve their daily lives.

JMIR Form Res 2022;6(7):e36018

doi:10.2196/36018

Keywords



Psychological distress, which can be broadly understood as an emotional state of suffering from mood- and anxiety-related symptoms [1], is among the leading causes of disability in the world [2]. It has become highly prevalent; it was recently estimated that up to 50% of the world's population experiences symptoms of psychological distress [3]. Knowing how widespread mental health issues are, it is not surprising that they were found to be a central cause for the global burden of diseases worldwide, accounting for some 56.7% of disability-adjusted life years [2]. Furthermore, a recent report estimated that in 2010, poor mental health resulted in a US $2-$5 trillion annual loss worldwide owing to poor health and reduced productivity. This loss was only projected to increase, expected to approach US $6 trillion by 2030 [4].

While the COVID-19 pandemic imposed numerous hardships on the world, the increase in prevalence and severity of mental distress symptoms reached a level where worldwide organizations declared it a global crisis that needed immediate attention [4,5]. While the number of individuals with mental illnesses has been documented in the second decade of the 21st century, there was no comparable increase in the numbers of individuals receiving treatment for these symptoms over the same period. This highlights the unmet needs for more treatment possibilities [6]. Furthermore, it was recently claimed that despite the substantial investment in the development of new medications and therapeutic protocols, they are only expected to have a limited impact over the long term. In contrast, it is argued that greater attention should be given to the development of prevention strategies and tools [7].

Recently, mobile apps and computer-based applications have become increasingly useful tools for creating and disseminating mental health prevention and treatment programs [8-10]. Mobile interventions introduce a fresh variety of possibilities, including the provision for unlimited interactions and psychological interventions. This creates opportunities to reach patients anytime and anywhere. Furthermore, this kind of platform also has the possibility to enhance efficiency by increasing the treatment intensity and integrating therapeutic strategies into daily life [11].

The use of mobile-based innervations was found to be particularly effective when it was used to deliver Acceptance Commitment Therapy (ACT) protocols [12-15]. Since ACT protocols allow for a low-intensity or high-frequency approach [12], they are a highly effective method to facilitate changes in the flexibility of the thinking process, which is one of ACT’s central features [16]. ACT is a form of psychotherapy positioned within the third wave of behavioral therapies. It focuses on teaching patients how to deal with challenging experiences by reflecting on their values [16]. ACT works to develop psychological flexibility, which entails having the ability to stay in contact with the present regardless of any unpleasant thoughts, feelings, or bodily sensations. Recently, ACT was shown to be effective in treating symptoms of both depression [14] and anxiety [13].

In this pragmatic retrospective study, we report the test results of a mobile platform called Kai.ai. Kai.ai is a platform designed to provide ACT-based, artificial intelligence–driven conversational coaching that helps users build habits for healthy living and resilience. It integrates seamlessly with all the top messaging apps (iMessage, WhatsApp, Discord, Telegram, etc) allowing for ecological momentary assessment and ecological momentary interventions. The platform checks in with users throughout the day and leverages users’ responses and journaling exercises to deliver tailored ACT skilled coaching. In this study, the platform was evaluated using both a pre-post and longitudinal design to assess changes in measures of well-being among users. Our hypothesis being that using Kai.ai will result in improved well-being.


Study Design

A pragmatic retrospective design was used for this study, using data collected as part of users’ engagement with Kai.ai’s services. All participants were located in the United States and had free access to the service, which is seamlessly integrated with all the top messaging apps (iMessage, WhatsApp, Discord, Telegram, etc). This retrospective study included data from 2909 Kai.ai users, 85% of whom were recruited from major social networks such as TikTok, and they are presented with an advertisement inviting them to test how happy they are. Clicking the advertisement would direct users to a landing page (How happy are you? - Kai), where they can complete the World Health Organization-Five Well-Being Index (WHO-5). Upon completing the questionnaire, users were asked whether they would like to continue and engage with Kai.ai in the future.

During the first engagement, users were asked to complete an electronically secured version of the WHO-5. The survey was completely optional, and users could opt to not take part in the assessment phase. As users continued using Kai.ai, they were prompted once every 6 weeks to complete an additional WHO-5 assessment.

Participants

Participants completed the WHO-5 at least twice, with the highest number of completed WHO-5 measurements being 9. All participants in this study were from the United States, but since the data set used in this study was completely anonymized before it was given to the authors, no gender- or age-related information was available.

Ethical Considerations

This report is based on a postfactum analysis, which used anonymized data gathered during users’ engagement with Kai.ai’s services. As such, it was determined as not being a study involving human subjects by WCG IRB (1-1504102-1).

WHO-5

The WHO-5 is a short self-report to measure well-being [17,18]. Items are rated on a 6-point Likert scale, where participants are asked to rate their experiences in the past 2 weeks from 5=all of the time to 0=at no time. Raw scores are then multiplied by 4 to transform the scale to a percentage-based scale, ranging from 0% to 100%. A transferred score of below 50% reflects poor well-being, and a 10-point change in the translated score is seen as clinically relevant [17]. Owing to its ease of use and high validity, the WHO-5 has been extensively used in mental health– and physical health–related mobile apps [19-23].

Intervention

Overview

Users can choose to interact with Kai.ai through common messaging apps. Each user’s first interaction starts with an onboarding process where the WHO-5 is administered. Based on the users’ responses, Kai.ai builds 3 routines for the morning, noon, and evening. Each routine includes one or more of the service’s main blocks: (1) Reflections and Journaling and (2) Exercises.

Reflections and Journaling

For Reflections and Journaling, users are invited to think and write about their experiences. To maintain structure and help guide users, Kai.ai uses three kinds of journaling cues: gratitude, learning, and one good thing that happened.

During the gratitude exercise, users are encouraged to identify things for which they are grateful and to list them in writing, where they can easily review this list daily—this gives room to develop flexible thinking patterns. In the learning exercise, users are asked to focus on the lessons they can learn from their experiences and, lastly, in one good thing that Exercises users learn to reduce stress and anxiety by focusing on a single task rather than a long to-do list and spreading themselves too thin. Adopting a routine reflection and journaling process is meant to help users achieve a more centered, grounded, joyous, and purposeful state of mind.

Exercises

Two main types of exercises are offered to users: breathing exercises and ACT training. The breathing exercises focus on teaching users how to breathe in a constructive manner; breathing through their noses, using their diaphragm, and noting their posture. These are all meant to ensure a better flow of oxygen to the body, allowing for a reduction in stress and anxiety through the operation of the parasympathetic nervous system [24].

The used ACT training is based on the mindfulness approach aimed at increasing psychological flexibility, and treating pain and discomfort as facts of life can be used for personal growth through a process of acceptance and validation [16].

User Interaction

Users have 2 main ways to interact with Kai.ai: passive and active. In the passive way, the service checks in on the user at least once and not more than 3 times a day, depending on the routine created during onboarding. The check-in prompts can include either Reflection and Journaling open-ended questions or closed questions where users are asked to indicate how happy they are. In addition to the passive check-ins, users can actively reach out to Kai.ai by simply texting it whenever they feel the need to.

Data Analyses

To test Kai.ai’s impact on participants’ well-being, 2 analytical methods were used. First, a paired samples t test was performed to compare participants’ baseline WHO-5 scores, which were collected just before the initiation of use, and the last WHO-5 score recorded for each participant. Following the initial analysis, hierarchical linear modeling (HLM) was used to examine the change in the users’ well-being over time. HLM was chosen over other repeated measures models, such as repeated measures ANOVA, owing to its unique ability to model incomplete or imbalanced data sets. In the first level, the HLM was modeled to estimate the individual linear slopes and intercepts, and in the second level, the sample’s average slope and intercept were estimated.


Paired Samples t Tests

Users' scores on the WHO-5 were compared from the first and last measurement of each user. On average, users’ last measurement score was better (median 52) than their first (median 40). A Wilcoxon signed rank test indicated that this difference was significant (W=2682927; P<.001). Figure 1 shows a box plot of the median (IQR) values of change in WHO-5 scores.

Figure 1. Box plot of the median (IQR) values of change in World Health Organization-Five Well-Being Index (WHO-5) scores for users' first and last measurements.
View this figure

HLMs

In addition to the paired samples t test that was used to compare users first and last measurement scores, an HLM was used since it is best suited to deal with incomplete data sets while still accounting for the data set’s repeated measures nature [25]. This unique strength of the HLM analysis was this analysis because the number of measurements among users varied between 2 and 9. Some users used the app for only 2 months, others used it for well over a year.

Consequently, our model tested whether differences in users’ WHO-5 score between any 2 consecutive measurements could be predicted by the level of engagement during the time frame. As such, we used 2 variables as indications of engagement: the number of unique days during which the user had active interactions with Kai.ai during the specified time frame and the number of daily messages the user sent during that time. Using the R nlme package [26], the effect of the engagement variables on the difference in WHO-5 scores was first tested separately for each measurement time point (second to ninth) and then aggregated over all time points. Our results show that the difference in WHO-5 scores was linearly related to the number of daily messages a user sent (β=.029; t81.36=4; P<.001) and the interaction between the number of messages and unique number of days (β=–.0003; t81.36=–2.2; P=.03), but not to the unique number of days on its own (Table 1)

Table 1. Unconditional growth model for changes in the World Health Organization-Five Well-Being Index (WHO-5) scores over time.
WHO-5 scoreEstimate (β)SEt test (df)P value
Intercept.5631.0450.536 (81.36).59
Total daily messages.029.0074 (81.36)<.001
Unique active days–.015.036–0.427 (81.36).67
Interaction–.0003.0001–2.2 (81.36) .03

Principal Findings

In this paper, we report the results of a pragmatic retrospective study aiming to test the effectiveness of a mobile phone–delivered, ACT-based, artificial intelligence–delivered conversational coaching platform. In the onboarding measurement, half of all participants reported a WHO-5 score of 40 or less, which is well below the cut-off point of 50, and an indicator of poor well-being. However, on the last measurement recorded for each user, half of all participants indicated a score of 52 and above, indicating a change for the better and an overall good well-being. These results highlight the great potential ACT-based mobile apps can have for improving users’ daily well-being.

Limitations

This paper describes the use of a novel platform and treatment protocol using mobile messaging apps to deploy ACT-based treatment to improve the well-being of users. Our results, while highlighting Kai.ai’s great potential as reflected by the pre-post results, also hints at the wealth of possibilities still left to discover. The results of our HLM model, especially those of the negative interaction between the engagement variables, indicates that further research is needed to fully understand and describe Kai.ai’s potential and fit. An example of such an area, which future research should expand on, is Kai.ai’s suitability to different genders and age groups. A recent study found differences among these groups in the adaptability of mobile-based coaching tools, as females and younger age groups (18-25 years) showed higher adaptability than males and older ones, respectively [27]. To overcome these adaptability challenges, a number of key features are introduced, which are directly aimed at improving users’ engagement, leveraging the motivations of choice and progression. As such, an emphasis was placed on greater choice, as users are given the option of choosing tracks that focus on different central core ideas, such as relationships, mood, and life habits. Furthermore, over time, the daily routines and exercises offered to the user are selected on the basis of past preferences. Lastly, by allowing users to review their progress, allowing users the ability to view their progress time.

Conclusions

The findings reported in this paper show Kai.ai’s great promise in helping individuals improve and maintain high levels of well-being and thus improve their daily life. These findings are further emphasized by the great need for mental health interventions, while the availability of and accessibility to treatment remains limited. It seems that the use of such technology, which is found to improve mental well-being per the findings of the WHO-5, is essential, especially nowadays.

Conflicts of Interest

AF is employed by Kai.ai, receives income from Kai.ai, and has been granted equity in Kai.ai. NN and MW receive income from Kai.ai and have been granted equity in Kai.ai.

  1. Drapeau A, Marchand A, Beaulieu-Prévost D. Epidemiology of Psychological Distress. In: L'Abate L, editor. Mental Illnesses - Understanding, Prediction and Control. Norderstedt: BoD – Books on Demand; 2012.
  2. Whiteford HA, Ferrari AJ, Degenhardt L, Feigin V, Vos T. The global burden of mental, neurological and substance use disorders: an analysis from the Global Burden of Disease Study 2010. PLoS One 2015;10(2):e0116820 [FREE Full text] [CrossRef] [Medline]
  3. Nochaiwong S, Ruengorn C, Thavorn K, Hutton B, Awiphan R, Phosuya C, et al. Global prevalence of mental health issues among the general population during the coronavirus disease-2019 pandemic: a systematic review and meta-analysis. Sci Rep 2021 May 13;11(1):10173 [FREE Full text] [CrossRef] [Medline]
  4. The Lancet Global Health. Mental health matters. Lancet Glob Health 2020 Nov;8(11):e1352 [FREE Full text] [CrossRef] [Medline]
  5. United Nations. COVID-19 and the Need for Action on Mental Health. UN Executive Office of the Secretary-General (EOSG) Policy Briefs and Papers. 2020 May 21.   URL: https://www.un-ilibrary.org/content/papers/27082245/9 [accessed 2022-05-25]
  6. Greenberg PE, Fournier A, Sisitsky T, Simes M, Berman R, Koenigsberg SH, et al. The Economic Burden of Adults with Major Depressive Disorder in the United States (2010 and 2018). Pharmacoeconomics 2021 Jun;39(6):653-665 [FREE Full text] [CrossRef] [Medline]
  7. Ormel J, Cuijpers P, Jorm A, Schoevers RA. What is needed to eradicate the depression epidemic, and why. Ment Health Prev 2020 Mar;17:200177. [CrossRef]
  8. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013 Nov 15;15(11):e247 [FREE Full text] [CrossRef] [Medline]
  9. Lindhiem O, Bennett C, Rosen D, Silk J. Mobile technology boosts the effectiveness of psychotherapy and behavioral interventions: a meta-analysis. Behav Modif 2015 Nov;39(6):785-804 [FREE Full text] [CrossRef] [Medline]
  10. Luxton D, McCann R, Bush N, Mishkind M, Reger G. mHealth for mental health: Integrating smartphone technology in behavioral healthcare. Prof Psychol Res Pr 2011 Dec;42(6):505-512. [CrossRef]
  11. Ebert D, Van Daele T, Nordgreen T, Karekla M, Compare A, Zarbo C, et al. Internet- and Mobile-Based Psychological Interventions: Applications, Efficacy, and Potential for Improving Mental Health. Eur Psychol 2018 May;23(2):167-187. [CrossRef]
  12. Krafft J, Potts S, Schoendorff B, Levin ME. A Randomized Controlled Trial of Multiple Versions of an Acceptance and Commitment Therapy Matrix App for Well-Being. Behav Modif 2019 Mar;43(2):246-272. [CrossRef] [Medline]
  13. Landy L, Schneider R, Arch J. Acceptance and commitment therapy for the treatment of anxiety disorders: a concise review. Curr Opin Psychol 2015 Apr;2:70-74. [CrossRef]
  14. Moore C, Hill J, Feinstein C, Pike C, Delaney K. Implementation of Acceptance Commitment Therapy on a Mood Disorder Unit: A Quality Improvement Project. J Psychosoc Nurs Ment Health Serv 2019 Nov 01;57(11):22-27. [CrossRef] [Medline]
  15. Reyes A. The Process of Learning Mindfulness and Acceptance through the Use of a Mobile App Based on Acceptance and Commitment Therapy: A Grounded Theory Analysis. Issues Ment Health Nurs 2022 Jan;43(1):3-12. [CrossRef] [Medline]
  16. Hayes SC. Acceptance and Commitment Therapy, Relational Frame Theory, and the Third Wave of Behavioral and Cognitive Therapies - Republished Article. Behav Ther 2016 Nov;47(6):869-885. [CrossRef] [Medline]
  17. Topp CW, Østergaard SD, Søndergaard S, Bech P. The WHO-5 Well-Being Index: a systematic review of the literature. Psychother Psychosom 2015;84(3):167-176 [FREE Full text] [CrossRef] [Medline]
  18. A. Mastering Depression in Primary Care. Hilleroed: World Health Organization, Regional Office for Europe, Psychiatric Research Unit, Frederiksborg General Hospital; 1998.
  19. Andreasson K, Krogh J, Bech P, Frandsen H, Buus N, Stanley B, et al. MYPLAN -mobile phone application to manage crisis of persons at risk of suicide: study protocol for a randomized controlled trial. Trials 2017 Apr 11;18(1):171 [FREE Full text] [CrossRef] [Medline]
  20. Coelhoso C, Tobo P, Lacerda S, Lima A, Barrichello C, Amaro JE, et al. A New Mental Health Mobile App for Well-Being and Stress Reduction in Working Women: Randomized Controlled Trial. J Med Internet Res 2019 Nov 07;21(11):e14269 [FREE Full text] [CrossRef] [Medline]
  21. Laird E, Ryan A, McCauley C, Bond R, Mulvenna M, Curran K, et al. Using Mobile Technology to Provide Personalized Reminiscence for People Living With Dementia and Their Carers: Appraisal of Outcomes From a Quasi-Experimental Study. JMIR Ment Health 2018 Sep 11;5(3):e57 [FREE Full text] [CrossRef] [Medline]
  22. Litvin S, Saunders R, Maier M, Lüttke S. Gamification as an approach to improve resilience and reduce attrition in mobile mental health interventions: A randomized controlled trial. PLoS One 2020;15(9):e0237220 [FREE Full text] [CrossRef] [Medline]
  23. Stallman H. Efficacy of the My Coping Plan mobile application in reducing distress: A randomised controlled trial. Clin Psychol 2020 Nov 09;23(3):206-212. [CrossRef]
  24. Jerath R, Edry J, Barnes V, Jerath V. Physiology of long pranayamic breathing: neural respiratory elements may provide a mechanism that explains how slow deep breathing shifts the autonomic nervous system. Med Hypotheses 2006;67(3):566-571. [CrossRef] [Medline]
  25. Maas C, Snijders T. The multilevel approach to repeated measures for complete and incomplete data. Qual Quant 2003;37(1):89. [CrossRef]
  26. Pinheiro J, Bates D, DebRoy S, Sarkar D, EISPACK, Heisterkamp S, R Core Team. nlme: Linear and Nonlinear Mixed Effects Models.   URL: https://cran.r-project.org/web/packages/nlme/nlme.pdf [accessed 2022-05-25]
  27. Lungu A, Jun JJ, Azarmanesh O, Leykin Y, Chen CE. Blended Care-Cognitive Behavioral Therapy for Depression and Anxiety in Real-World Settings: Pragmatic Retrospective Study. J Med Internet Res 2020 Jul 06;22(7):e18723 [FREE Full text] [CrossRef] [Medline]


ACT: Acceptance Commitment Therapy
HLM: hierarchical linear modeling
WHO-5: World Health Organization-Five Well-Being Index


Edited by A Mavragani; submitted 28.12.21; peer-reviewed by A Hashim; comments to author 26.01.22; revised version received 15.03.22; accepted 20.05.22; published 12.07.22

Copyright

©Navot Naor, Alex Frenkel, Mirène Winsberg. Originally published in JMIR Formative Research (https://formative.jmir.org), 12.07.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.