A pragmatic retrospective design was used for this study, using data collected as part of users’ engagement with Kai.ai’s services. All participants were located in the United States and had free access to the service, which is seamlessly integrated with all the top messaging apps (iMessage, WhatsApp, Discord, Telegram, etc). This retrospective study included data from 2909 Kai.ai users, 85% of whom were recruited from major social networks such as TikTok, and they are presented with an advertisement inviting them to test how happy they are. Clicking the advertisement would direct users to a landing page (How happy are you? - Kai), where they can complete the World Health Organization-Five Well-Being Index (WHO-5). Upon completing the questionnaire, users were asked whether they would like to continue and engage with Kai.ai in the future.

During the first engagement, users were asked to complete an electronically secured version of the WHO-5. The survey was completely optional, and users could opt to not take part in the assessment phase. As users continued using Kai.ai, they were prompted once every 6 weeks to complete an additional WHO-5 assessment.

Participants completed the WHO-5 at least twice, with the highest number of completed WHO-5 measurements being 9. All participants in this study were from the United States, but since the data set used in this study was completely anonymized before it was given to the authors, no gender- or age-related information was available.

This report is based on a postfactum analysis, which used anonymized data gathered during users’ engagement with Kai.ai’s services. As such, it was determined as not being a study involving human subjects by WCG IRB (1-1504102-1).

The WHO-5 is a short self-report to measure well-being [17,18]. Items are rated on a 6-point Likert scale, where participants are asked to rate their experiences in the past 2 weeks from 5=all of the time to 0=at no time. Raw scores are then multiplied by 4 to transform the scale to a percentage-based scale, ranging from 0% to 100%. A transferred score of below 50% reflects poor well-being, and a 10-point change in the translated score is seen as clinically relevant [17]. Owing to its ease of use and high validity, the WHO-5 has been extensively used in mental health– and physical health–related mobile apps [19-23].

Users have 2 main ways to interact with Kai.ai: passive and active. In the passive way, the service checks in on the user at least once and not more than 3 times a day, depending on the routine created during onboarding. The check-in prompts can include either Reflection and Journaling open-ended questions or closed questions where users are asked to indicate how happy they are. In addition to the passive check-ins, users can actively reach out to Kai.ai by simply texting it whenever they feel the need to.

To test Kai.ai’s impact on participants’ well-being, 2 analytical methods were used. First, a paired samples t test was performed to compare participants’ baseline WHO-5 scores, which were collected just before the initiation of use, and the last WHO-5 score recorded for each participant. Following the initial analysis, hierarchical linear modeling (HLM) was used to examine the change in the users’ well-being over time. HLM was chosen over other repeated measures models, such as repeated measures ANOVA, owing to its unique ability to model incomplete or imbalanced data sets. In the first level, the HLM was modeled to estimate the individual linear slopes and intercepts, and in the second level, the sample’s average slope and intercept were estimated.

Users' scores on the WHO-5 were compared from the first and last measurement of each user. On average, users’ last measurement score was better (median 52) than their first (median 40). A Wilcoxon signed rank test indicated that this difference was significant (W=2682927; P<.001). Figure 1 shows a box plot of the median (IQR) values of change in WHO-5 scores.

In addition to the paired samples t test that was used to compare users first and last measurement scores, an HLM was used since it is best suited to deal with incomplete data sets while still accounting for the data set’s repeated measures nature [25]. This unique strength of the HLM analysis was this analysis because the number of measurements among users varied between 2 and 9. Some users used the app for only 2 months, others used it for well over a year.

Consequently, our model tested whether differences in users’ WHO-5 score between any 2 consecutive measurements could be predicted by the level of engagement during the time frame. As such, we used 2 variables as indications of engagement: the number of unique days during which the user had active interactions with Kai.ai during the specified time frame and the number of daily messages the user sent during that time. Using the R nlme package [26], the effect of the engagement variables on the difference in WHO-5 scores was first tested separately for each measurement time point (second to ninth) and then aggregated over all time points. Our results show that the difference in WHO-5 scores was linearly related to the number of daily messages a user sent (β=.029; t_81.36=4; P<.001) and the interaction between the number of messages and unique number of days (β=–.0003; t_81.36=–2.2; P=.03), but not to the unique number of days on its own (Table 1)

In this paper, we report the results of a pragmatic retrospective study aiming to test the effectiveness of a mobile phone–delivered, ACT-based, artificial intelligence–delivered conversational coaching platform. In the onboarding measurement, half of all participants reported a WHO-5 score of 40 or less, which is well below the cut-off point of 50, and an indicator of poor well-being. However, on the last measurement recorded for each user, half of all participants indicated a score of 52 and above, indicating a change for the better and an overall good well-being. These results highlight the great potential ACT-based mobile apps can have for improving users’ daily well-being.

This paper describes the use of a novel platform and treatment protocol using mobile messaging apps to deploy ACT-based treatment to improve the well-being of users. Our results, while highlighting Kai.ai’s great potential as reflected by the pre-post results, also hints at the wealth of possibilities still left to discover. The results of our HLM model, especially those of the negative interaction between the engagement variables, indicates that further research is needed to fully understand and describe Kai.ai’s potential and fit. An example of such an area, which future research should expand on, is Kai.ai’s suitability to different genders and age groups. A recent study found differences among these groups in the adaptability of mobile-based coaching tools, as females and younger age groups (18-25 years) showed higher adaptability than males and older ones, respectively [27]. To overcome these adaptability challenges, a number of key features are introduced, which are directly aimed at improving users’ engagement, leveraging the motivations of choice and progression. As such, an emphasis was placed on greater choice, as users are given the option of choosing tracks that focus on different central core ideas, such as relationships, mood, and life habits. Furthermore, over time, the daily routines and exercises offered to the user are selected on the basis of past preferences. Lastly, by allowing users to review their progress, allowing users the ability to view their progress time.

The findings reported in this paper show Kai.ai’s great promise in helping individuals improve and maintain high levels of well-being and thus improve their daily life. These findings are further emphasized by the great need for mental health interventions, while the availability of and accessibility to treatment remains limited. It seems that the use of such technology, which is found to improve mental well-being per the findings of the WHO-5, is essential, especially nowadays.