Depression, anxiety, and loneliness remain highly prevalent worldwide, yet many individuals do not receive adequate psychological support due to stigma, cost, and limited access [1]. Even among those who receive care, fewer than half benefit substantially from current interventions, underscoring the need for scalable mental health solutions [2-5].

Conversational AI (CAI) tools have been studied as scalable approaches to mental health support and self-management of depression, anxiety, and well-being [6-8]. Advances in speech recognition, natural language processing (NLP), and machine learning allow conversational agents (CAs) to assess clinical information and adapt interventions [9]. However, limited empathy, inconsistent alliance, and high dropout remain concerns [10-12].

Recent CAI systems are increasingly powered by large language models (LLMs), which can be adapted or fine-tuned for specific applications, including mental health support. Multimodal CAI systems that integrate both text-based and voice-based interactions may help address these limitations by fostering stronger engagement and therapeutic alliance [8,13,14]. Emerging work on “synthetic relationships” similarly suggests that continuous, adaptive interactions may support therapeutic processes [15], a critical factor for effective psychological treatment [13,16].

Evidence from naturalistic, voice-based use remains sparse [10], and little is known about the persistence of the perceived alliance. Prior studies have reported associations between baseline symptom severity or loneliness and subsequent outcomes in AI-supported interventions [17-20].

Relational characteristics, including attachment anxiety, epistemic trust, fear of intimacy, and personality traits, have also been associated with willingness to adopt CAI-based support [21-23]. These findings raise questions about how selection into CAI use relates to sustained engagement and observed outcomes [17].

A prior baseline study of 527 self-referred users of the Clare mental health chatbot addressed part of this question by showing that individuals often engage with CAs under conditions of considerable emotional and social vulnerability. That baseline report focused on cross-sectional data, describing participants’ motives, expectations, and bonding with Clare (Working Alliance Inventory–Short Revised [WAI-SR]), as well as a broad set of psychosocial and demographic variables. Most users reported elevated loneliness (86%), anxiety (69%), and depression (59%), and many cited avoiding embarrassment or concerns about appearance in face-to-face therapy as reasons for choosing a chatbot. Users also reported a strong perceived working alliance with Clare (WAI-SR; mean 3.76, SD 0.72) [14].

The baseline analysis did not examine longitudinal interaction patterns, perceived alliance, or psychological measures. The present study uses the same cohort to address these questions [14].

This exploratory feasibility study described engagement, perceived alliance, and psychological measures over 4 to 8 weeks of real-world Clare use. It characterized modality preferences and explored associations between baseline psychological profiles, subsequent engagement, alliance, and symptom scores. The analyses were descriptive and hypothesis-generating.

Clare is an AI-based CA developed by clare&me GmbH, now owned by HealthyCloud GmbH. The evaluated version was deployed during the study’s data collection period (November 2023 to April 2024) and combined a symptom-to-intervention matching algorithm, proprietary NLP-based safeguarding mechanisms for mental health use, and a self-hosted, fine-tuned open-source Llama LLM.

The LLM was trained on a synthetic data set iteratively developed and reviewed by psychologists and psychotherapists, in collaboration with licensed clinicians, to translate evidence-based cognitive behavioral therapy (CBT) elements into supportive conversations. With user consent, conversational data and derived recommendations may be stored to enable personalized support via a selective memory function [24]. Clare is intended to complement, not replace, professional mental health care and does not provide diagnosis or emergency services.

Minor software updates during the study did not affect the underlying LLM, therapeutic content, or safeguarding mechanisms; further technical details cannot be disclosed for proprietary reasons.

Clare provides psychological support through voice-based and text-based conversations grounded in CBT, mindfulness, and self-compassion. CBT is a structured, evidence-based psychotherapeutic approach that targets cognitive and behavioral processes through techniques such as Socratic questioning and cognitive restructuring [14,24].

Interactions followed a semistructured conversational flow—an emotional check-in, identification of current concerns, and delivery of targeted interventions (eg, breathing exercises, cognitive restructuring, psychoeducation, or sleep hygiene), and users could engage on demand via phone or text, switching flexibly between modalities (Figure 1).

Intervention content was selected through a rule-based system developed by clinical psychologists, mapping user input (eg, expressions of stress or fatigue) to tailored coping strategies and adapting responses based on interaction history. Voice interactions tended to be more guided and linear, while text interactions allowed greater flexibility and user-directed pacing; sessions typically lasted 5 to 45 minutes depending on engagement and intervention type.

All participants completed an onboarding call covering Clare’s purpose, scope, limitations, and instructions for use, and were informed that it was an AI-based CA—not a human therapist or substitute for psychotherapy—and not intended for emergency or crisis care. Participants could engage with Clare on demand via phone or text, switch flexibly between modalities, and use it for as long as they wished (Figure 1). Registration and onboarding initiation constituted enrollment, while exposure was defined as any interaction beyond the onboarding call; inclusion in the analytic samples required completing the onboarding call and relevant assessments, plus at least 1 interaction between assessment time points. Interaction metrics captured the frequency and duration of calls and text-based interactions.

The system incorporates rule-based safety mechanisms, content filtering, and automated risk assessment to identify safety-relevant content, including suicidality, self-harm, harm to others, and elevated clinical symptom burden. When risk indicators are detected, the system may initiate clarifying questions to assess severity before providing an empathic response, communicating the limitations of AI support, and directing users to appropriate professional services and crisis resources in their country. These safety protocols were developed and reviewed in collaboration with practicing psychologists to support the ethical management of risk-related disclosures.

Conversations were not monitored by clinicians in real time; instead, risk detection and response were managed through the automated safety system. Conversation data could be reviewed retrospectively for quality assurance, safety evaluation, and system improvement in accordance with user consent. Further details on Clare’s design and safety framework are reported in Schäfer et al [14].

This naturalistic, single-arm, repeated-measures observational study examined real-world use of Clare among self-referred users, drawing on data from the Clare baseline study. Participants independently signed up and engaged with the intervention outside a controlled research setting. The primary aim was to explore engagement patterns, feasibility, and preliminary associations rather than evaluating causal efficacy. Participants interacted with the multimodal CA, which supported both voice and text communication, over an 8-week period.

Data were collected via SoSciSurvey between November 20, 2023, and April 21, 2024. Assessments occurred at sign-up prior to intervention use (T1), 3 to 5 days after initial engagement (T1b), week 4 (T2), and week 8 (T3); participants could complete questionnaires after the scheduled date, so T2 and T3 refer to the scheduled rather than actual completion time points. Because registration did not necessarily indicate active use, retention and longitudinal analyses were restricted to participants meeting engagement criteria established in the baseline study.

Participants were recruited between November 2023 and April 2024 through English-language advertisements on Facebook, Instagram, LinkedIn, and Google Ads, using language targeting based on users’ browser or account settings. No demographic targeting was applied. The sampling frame comprised adults interested in using a digital mental health intervention who had access to a mobile or internet-enabled device.

Participants received detailed information on the study and completed the baseline survey (T1) via SoSciSurvey after providing informed consent. Eligible participants were aged 18 to 65 years, fluent in English, and had access to a mobile internet-enabled device. Detailed baseline characteristics of this cohort have been reported previously [14].

To ensure meaningful exposure to the intervention, analyses were restricted to participants who progressed beyond enrollment and actively engaged with Clare. Enrollment was defined as registration and initiation of the onboarding process, whereas active engagement required completion of the first postengagement assessment (T1b) and at least one voice- or chat-based interaction between assessment periods. Analyses further included only participants who met the predefined engagement criteria and did not report suicidality at onboarding or during follow-up, as described previously [14]. No formal sample size calculation was conducted; sample size was determined by feasibility within the recruitment period.

Two analytic samples were defined. The 4-week sample (T2; n=53) included participants who completed T1b and T2. The 8-week sample (T3; n=21) included participants who completed all assessments from T1 to T3. Based on the engagement criteria, participants were required to complete at least two interactions for inclusion in the 4-week sample and at least three interactions for inclusion in the 8-week sample, in addition to the onboarding call. The analytic samples therefore represent a subset of enrolled and actively engaged participants and may not be representative of all registered users. Further details are provided in Multimedia Appendix 1.

In addition to the automated safety mechanisms described, anonymized conversation transcripts were reviewed twice weekly by 2 members of the research team for quality assurance and safety monitoring. Reviews focused on identifying potential failures of risk detection, inappropriate or clinically inaccurate responses, and other safety-relevant issues. Suicidality detection and sensitive-topic dialogue management were implemented as dedicated, rule-governed components.

Safety monitoring supported ongoing system refinement; however, formal evaluation of safety performance metrics (eg, false-positive and false-negative rates) was not performed as part of this observational study.

The technical design was rooted in responsible and ethical guidelines on the development of AI in mental health [25,26] and psychological practice [27], determining when the system should de-escalate or redirect users to human assistance.

The study followed an observational design without randomization or a control group. The study protocol was approved by the Ethics Committee of Campus Charité Mitte, Berlin, Germany (EA1/109/22), and the procedures complied with applicable institutional and national ethical standards. Study data were collected and managed independently by the Charité research team; HealthyCloud GmbH did not have access to the deidentified research data set. All participants provided informed consent prior to participation. Participant data were deidentified before analysis and handled in accordance with applicable data protection regulations. Participants received €15 (€1=US $1.07 as of April 19, 2024) as compensation upon completion of the survey and were allowed to continue using Clare free of charge after the study.

While newer approaches such as ecological momentary assessment (EMA) [28,29] offer real-time symptom assessment and could be integrated with AI-based interventions, we used traditional self-report measures to ensure consistency with prior digital mental health research.

Outcomes included psychological distress (Patient Health Questionnaire–4 [PHQ-4] and Patient Health Questionnaire–Distress [PHQ-D], including the stress module), perceived working alliance (WAI-SR), mobile therapeutic alliance (Mobile Agnew Relationship Measure [mARM]), loneliness (UCLA Loneliness Scale [UCLA]), life satisfaction (Satisfaction With Life Scale [SWLS]), and AI engagement metrics (call frequency, duration, and modality). All measures except the mARM were assessed at T1, T1b, T2 (week 4), and T3 (week 8), with T1b serving as the analytic baseline for outcomes requiring prior interaction; the mARM was assessed only at T2 and T3, reflecting the assumption that therapeutic relationship development requires sustained engagement.

Situational context variables were collected only at T3. We report depression, anxiety, distress, loneliness, and social anxiety, as these constructs are most commonly examined in AI mental health studies and allow for comparability with existing evidence [6,19,30,31]. Except for the mARM, all measures are described in detail in Schäfer et al [14].

Data were collected automatically through the web-based interface of the Clare platform during user interactions. These data included engagement metrics such as the number of calls, call duration, and modality of interaction (voice or text), as well as timestamps of usage. Engagement data (onboarding call initiation, texting, and calling) were captured via automated logs of user interactions with the system, without manual data entry by researchers. A more detailed description of the captured engagement data is available in our baseline study report [14].

At the 8-week assessment (T3), participants reported the circumstances in which they used Clare using predefined response options (eg, at home, on the go, during a crisis, as part of a routine, or at specific times of day). Participants were also asked whether they would recommend Clare for self-help purposes. In addition, an optional open-ended question invited participants to provide feedback regarding their experience with the intervention and study procedures. Open-ended responses were reviewed descriptively and summarized narratively; no formal qualitative analysis was conducted.

Analyses examined engagement patterns and changes in mental health and bonding measures across T1, T1b, T2, and T3. Two analytic subsamples were used: a 4-week follow-up sample (T2; n=53) and an 8-week follow-up sample (T3; n=21), both consisting of participants who met the predefined engagement and assessment completion criteria. Categorical engagement data were summarized using frequencies; open-ended responses were summarized narratively. To examine interaction modality, participants were categorized based on their primary mode of engagement with the intervention. Descriptive statistics are presented in Table 1. “Voice-only” users engaged exclusively via phone calls, “text-only” users interacted exclusively through written messages, and “hybrid” users used a combination of both modalities. As the majority of participants engaged with the intervention using a hybrid modality, modality-specific descriptive analyses primarily focused on this group to maximize the number of observations available for interpretation.

Quantitative analyses were conducted in SPSS version 28. Paired samples 2-tailed t tests compared WAI-SR scores between T1b and T2 and between T1b and T3, and mARM scores between T2 and T3. For all other mental health outcomes, comparisons were made between T1 and T2 and between T1 and T3. Mean differences and 95% CIs were reported as estimation-based summaries, with Cohen d values and P values reported only in tables.

Retention was defined based on continued engagement with Clare rather than registration alone, as sign-up did not necessarily indicate active use of the intervention. Participant retention was tracked across 3 key time points: T1b (first engagement-based assessment), T2 (weeks 1‐4), and T3 (weeks 1‐8). T1b reflects an early postengagement assessment rather than the baseline assessment. Participant progression from baseline assessment (n=527) to enrollment (n=369), active engagement (n=348), and analytic inclusion at T2 (n=53) and T3 (n=21) is illustrated in Figure 2.

Given the exploratory nature of the study, no correction for multiple comparisons was applied. This approach was chosen to maintain sensitivity for detecting potentially meaningful associations and to reduce the risk of Type II errors, particularly given the limited sample size at follow-up.

Given the high attrition rates, primary analyses were based on complete cases. To assess the missing data mechanism, Little’s test for missing completely at random (MCAR) was performed. Because the data were not assumed to be MCAR, sensitivity analyses were conducted using linear mixed-effects models with restricted maximum likelihood. Models included time (T1, T2, and T3) as a fixed effect, a random intercept for participant, and a first-order autoregressive covariance structure to account for repeated observations within individuals. All available observations from participants contributing data at one or more assessment points were included; complete data across all waves were not required. The number of participants and observations contributing to each model varied by outcome due to attrition and outcome-specific missingness. No data imputation was performed.

To characterize attrition, baseline demographic, clinical, and alliance variables were compared between completers and noncompleters using the 2-tailed Welch t tests and descriptive statistics. The Welch t tests were selected because the assumption of homogeneity of variance was violated. Effect sizes were calculated using Cohen d (Multimedia Appendix 3).

Participant flow is shown in Figure 2. At T1b, 348 participants remained actively engaged. Of these, 53 (15.2%) met the T2 analytic criteria and 21 (6.0%) met the T3 criteria, corresponding to 94.0% attrition from T1b to T3.

Between-group estimates and 95% CIs for baseline differences between completers and noncompleters are provided in Multimedia Appendix 3.

No instances of self-harm or suicidality were identified by the automated risk assessment system during the study period, and no participants were excluded or withdrawn because of safety-related concerns. Transcript reviews did not identify any safety-critical system responses requiring participant follow-up or clinical intervention.

The T2 (n=53) and T3 (n=21) analytic samples were demographically similar. The mean ages of participants in the T2 (n=53) and T3 (n=21) analytic samples were 33.19 (SD 9.74; range 18-55) years and 34.86 (SD 10.65; range 19-55) years, respectively, and most were based in the United Kingdom or the United States. Sociodemographic characteristics across baseline (T1), early postengagement (T1b), 4-week (T2), and 8-week (T3) samples are listed in Table 2.

Table 2 displays baseline (T1) scores for depression, anxiety, distress, social anxiety, loneliness, and life satisfaction for the 4-week (n=53) and 8-week (n=21) analytic subsamples. Values fall within comparable ranges across groups and reflect moderate distress and social anxiety alongside relatively low life satisfaction.

Hybrid use was the most frequently recorded modality among retained participants, whereas voice-only use was rare (Table 3).

After 8 weeks, participants (n=21) indicated the contexts in which they used the CAI (multiple responses allowed). Most reported using it at home (18/21, 85.7%), whereas only 2 (9.5%) participants used it “on the go,” and none reported use during an acute crisis. Regarding timing, 9 (42.9%) participants used the CAI as part of a routine, whereas 11 (52.4%) used it flexibly. Of 21 participants, 8 (38.1%) used it during the day, and 5 (23.8%) at night. Most participants stated they would recommend Clare for self-help (18/21, 85.7%).

Written feedback was provided by 10 (47.6%) participants, with most comments expressing satisfaction with the CAI or the questionnaire (eg, “Everything was good,” “Very clear and no problems encountered,” and “I am satisfied with Clare”); some pointed to minor ambiguities (eg, distinguishing in-person from online therapy). The feedback received was generally positive; however, the limited number of comments and the absence of a formal qualitative analysis preclude firm conclusions regarding usability.

Engagement metrics across weeks 1 to 4 are presented in Table 4. Engagement was highest in week 1 and declined consistently across all indicators over the study period.

Changes in working alliance and mental health outcomes were examined for the 4-week (n=53) and 8-week (n=21) analytic samples; sample sizes varied slightly across outcomes due to missing data. Longitudinal trajectories are presented in Table 5 and Table 6. Additional analyses are provided in Multimedia Appendix 4 and Multimedia Appendix 3.

From T1b to T2 (n=53), mean WAI-SR total scores increased by 0.20 (95% CI −0.01 to 0.42) points. Bond scores at T2 exceeded the ≥3.45 threshold proposed by Jasper et al [41] for human therapeutic contexts. Among the retained 8-week subgroup (n=21), mARM scores increased by 0.22 points from T2 to T3 (95% CI 0.03 to 0.42; Table 5 and Table 6).

Among T1–T2 complete pairs (n=52), mean PHQ-D scores decreased by 1.13 (95% CI 0.25 to 2.02) points, PHQ-4 scores decreased by 0.69 (95% CI −0.01 to 1.40) points, and Mini-SPIN scores decreased by 1.73 (95% CI 0.88 to 2.58) points (Table 5).

Among T1-T2 complete pairs (n=52), mean SWLS scores increased by 6.23 (95% CI 4.70 to 7.76) points (Table 5).

Among T1–T2 complete pairs (n=52), mean UCLA scores increased by 0.48 (95% CI 0.00 to 0.96) points. Among the 8-week subgroup, the descriptive means were 5.81 at T1, 6.43 at T2, and 5.38 at T3 (Table 6).

Table 6 presents descriptive trajectories for the 21 participants retained through T3. These estimates describe this selected subgroup and do not estimate an intervention effect.

Associations between early communication frequency, alliance, and mental health measures at T2 were explored in the 4-week sample (n=53). Full estimates are provided in Multimedia Appendix 5.

Correlations between WAI-SR and depression, anxiety, loneliness, or social anxiety had absolute values of 0.20 or less. Message-volume correlations with WAI-SR ranged from ρ of −0.36 (approximate 95% CI −0.57 to −0.10) to ρ of −0.28 (approximate 95% CI −0.50 to −0.01); call-frequency and call-duration estimates were closer to zero (Multimedia Appendix 5).

Little’s MCAR assessment indicated that missingness was not completely at random. Sensitivity analyses using all available observations produced changes in the same direction as the primary complete-case analyses for WAI-SR, PHQ-D, Mini-SPIN, and SWLS, whereas the mARM estimate indicated little change. Thus, the overall pattern broadly converged with the primary analyses, although attrition-related bias remains possible; detailed model outputs are provided in Multimedia Appendix 6.

Sustained engagement was limited: 53 out of 348 (15.2%) actively engaged users met the 4-week criteria and 21 (6.0%) met the 8-week criteria. Hybrid voice-and-text use was common among retained participants, whereas voice-only use was rare. In the complete-case samples, mean alliance and several symptom or well-being scores changed over time, but wide or boundary-crossing CIs for several estimates, the absence of a control group, and the highly selected follow-up samples limit interpretation.

Higher baseline distress and loneliness were associated with lower subsequent engagement. Alliance estimates were only weakly associated with communication volume or symptom measures, indicating that perceived bonding and usage intensity should not be treated as interchangeable indicators. These findings describe associations among retained users and do not establish effectiveness, causal mechanisms, or the clinical importance of the observed changes.

In this uncontrolled naturalistic study, engagement declined substantially and only small, selected subgroups completed the 4-week and 8-week assessments. Hybrid use was common among those retained, and perceived alliance and several mental health scores changed among completers, but the estimates cannot be attributed to Clare. The divergence between usage and perceived alliance highlights the need for clearer, AI-appropriate definitions and measures of relational engagement. Controlled, adequately powered studies with stronger retention, systematic safety evaluation, and longer follow-up are needed.