Mental health conditions such as depression and anxiety remain among the leading causes of disability worldwide, yet access to timely, evidence-based care continues to fall short of population need [1]. Structural barriers such as high costs, shortages of trained clinicians, long waitlists, geographic disparities, and perceived stigma limit the reach of traditional psychotherapy and contribute to persistent treatment gaps [2,3]. As a result, scalable digital solutions have become a major focus for innovation, with particular attention to platforms that can deliver immediate, flexible, and low-threshold support outside of, or as a supplement to, formal clinical settings.

Large language model (LLM)–based chatbots are rapidly evolving in the digital mental health space [4]. As reviewed in recent narrative and systematic syntheses of artificial intelligence (AI) mental health chatbots [5,6], these chatbots offer the possibility of responsive, personalized conversational support delivered at scale, with emerging studies suggesting they may help reduce mental health symptoms, increase engagement, and broaden access to care [7-13]. Yet, some researchers have been critical of diving straight into clinical trials [14], as the development of these products has largely outpaced the empirical evidence base, with key questions remaining about how therapeutic processes unfold within chatbot interactions, how users perceive safety and trust, and whether these tools function best as stand-alone interventions or complements to clinician-delivered care [15,16].

To date, studies assessing chatbots as mental health tools provide limited transparency into how prototypes were designed, refined, and evaluated for safety. As a result, little is known about how users actually experience early-stage chatbots, such as how safe, helpful, or trustworthy they feel, what facilitates or impedes engagement, and how these factors affect the impact of the chatbot. This gap is consequential, as understanding users’ initial experiences with mental wellness chatbot prototypes is essential for responsible development and real-world effectiveness. Indeed, prior research on digital mental health interventions shows that early impressions such as perceived safety, personalization, and feeling understood are strongly associated with subsequent engagement, adherence, and indicators of therapeutic response [17,18]. These experiential factors also shape trust and therapeutic alliance, which are central determinants of whether users continue to engage long enough to experience meaningful benefits [19,20]. However, recent theoretical work suggests that alliance and other change processes in chatbot-based interventions may not operate identically to those in human psychotherapy, given that chatbots lack human therapists’ ontological and sociocultural status [21]. This further highlights the need to empirically examine how users experience and interpret interactions with early-stage mental wellness chatbots.

Whereas several recent studies have focused on symptom outcomes in randomized trials [5] or evaluated chatbot performance in controlled vignette-based simulations [22], fewer investigations have examined early-stage prototype development using mixed methods approaches that integrate user experience, safety workflows, and preliminary clinical signals. As a result, there remains limited visibility into how conversational design choices, early user impressions, and structured protocols shape engagement and short-term outcomes. Because complex, multidimensional user experiences are best captured using human-centered, mixed methods that combine quantitative usability and symptom metrics with qualitative insight into how users interpret and experience emerging AI tools [23], early-stage evaluations benefit from integrating both forms of evidence. This study provides a comprehensive assessment of an early-stage chatbot prototype, explicitly linking user experience and exploratory clinical outcomes to inform responsible refinement and progression to larger trials.

Situated within digital mental health evaluation research, this pilot study used mixed methods to evaluate an early-stage LLM–based mental wellness chatbot designed to support adults experiencing at least mild anxiety or depression. Across 3 successive cohorts, participants interacted with the chatbot over a 2-week period, providing both quantitative and qualitative feedback on usability, acceptability, and perceived therapeutic value. We also explored pre-post changes in mental health symptoms as a preliminary gauge for a potential therapeutic signal. In the absence of randomization or a comparison arm, early clinical signals from chatbot prototypes should be interpreted as exploratory rather than as evidence of efficacy [24]. By integrating experiential and early clinical outcome data within a structured pilot framework, this study contributes prototype-stage evidence to inform responsible refinement, optimization, and future controlled evaluation of mental wellness chatbots. Accordingly, the aim of this study was to conduct a comprehensive mixed methods evaluation of a generative LLM–based mental wellness chatbot, examining user experience, safety monitoring feasibility, and exploratory clinical signal to inform responsible prototype refinement and future controlled trials.

The LLM–powered mental wellness chatbot evaluated in this study was developed to provide brief emotional support for adults experiencing mild to moderate anxiety or depressive symptoms. Responses were guided by evidence-based principles drawn from supportive and cognitive-behavioral therapy frameworks, enabling the chatbot to respond empathically to user input while encouraging adaptive coping and reflection.

Rather than delivering a structured, manualized treatment protocol, the chatbot facilitates supportive mental wellness conversations informed by evidence-based common factors, including emotional validation and reflective listening, collaborative clarification of goals, and brief coping or activation prompts (eg, grounding or perspective-taking exercises) [25]. These processes were intentionally scaffolded through system-level instructions emphasizing emotional safety, nondirectiveness, and avoidance of clinical diagnosis or prescriptive advice. The app functioned as a stand-alone iOS application with a cloud-based backend hosted by Amazon Web Services. Additional details regarding the therapeutic framework, AI architecture, and real-time safety classifier are provided in the Multimedia Appendix 1 [26-37].

The study was conducted from September to November 2025. Participants were recruited through dscout (dscout Inc), a web-based research platform with a large opt-in panel of adults across the United States that is focused on user experience research. Eligible participants were aged ≥21 years who reported at least mild symptoms of anxiety or depression at the time of screening. Individuals who endorsed suicidal or self-harm thoughts in the past 2 weeks (ie, score of ≥1 on the ninth item of the Patient Health Questionnaire-9 [PHQ-9]) or serious mental illness, as well as those currently receiving psychotherapy, were excluded from this early-stage pilot test of an app that has not yet been evaluated for individuals with complex clinical needs. Additional screening and demographic balance procedures are described in the Multimedia Appendix 1 [26-37]. Enrollment occurred in 3 successive cohorts to allow iterative refinement of the chatbot prototype between cohorts. After completing eligibility screening, participants were invited to complete a baseline survey assessing demographics, clinical symptomatology (eg, depressive and anxiety symptoms), and attitudes toward mental health therapy and chatbots for emotional support. After baseline, participants were instructed to download the prototype application via TestFlight—Apple’s official beta-testing platform for iOS. Participants were instructed to use the chatbot for at least 60 minutes per week during the 2-week study period. The first use of the chatbot triggered a first impressions survey and open-ended qualitative prompts focused on initial user experience feedback. One week following the app download, participants completed a brief follow-up survey and qualitative prompts focused on usability and acceptability. Finally, the 2-week follow-up survey reassessed the clinical metrics and attitudes toward mental wellness chatbots originally assessed at baseline. All survey instruments/measures and interview materials are described in the Multimedia Appendix 1 [26-37].

Chatbot interactions were continuously monitored in real-time by an automated, transcript-based safety classifier that was calibrated to high sensitivity due to the early stage of chatbot testing. The classifier detects participant responses indicating potential harm to self, to others, or from others (Multimedia Appendix 1 [26-37] provides details). All flagged responses triggered a real-time alert to the study clinicians, who reviewed each flagged transcript within 2 hours to determine whether phone-based outreach for a safety check or emergency intervention was warranted.

The screening survey was hosted on dscout and distributed to their active panel of participants. As required by dscout, the study description made clear that this was a 2-week digital mental health and well-being study involving the use of a mobile app. The screener stopped participants as soon as they failed an eligibility criterion to reduce the burden of continuing to ask unnecessary items once ineligibility was established. Because exclusions occurred dynamically throughout the screener, precise counts for every individual exclusion step cannot be calculated.

A detailed description of eligibility screening and a participant flow diagram is provided in Figure S2 in Multimedia Appendix 1 [26-37]. In short, 3406 individuals completed the screening, and 184 (5.4%) were eligible, with the most common exclusions being living in an ineligible state, not meeting the minimum symptom severity threshold (PHQ-2 or Generalized Anxiety Disorder-2 [GAD-2] score ≥3), or being currently engaged in some form of mental health care. Of the eligible pool, 125 were ultimately enrolled. Consistent with the aims of pilot-stage testing, this sample was not designed to represent individuals with acute or high-risk clinical presentations. The final sample represented 32 different US states, ranging in age from 21 to 67 (mean 34.6, SD 9.9) years, and 80 (64%) identified as women. While most reported being White (74/125, 59.2%), 19 (15.2%) were Black or African American, 13 (10.4%) were Asian or Asian American, 10 (8%) were multiple races, and 9 (7.2%) reported another race or preferred not to answer. Regarding ethnicity, 13 (10.4%) were Hispanic/Latinx. Most were employed (91/125, 72.8%), while 17 (13.6%) listed their employment status as currently a student, and 17 (13.6%) were not employed. More than half (70/125, 56%) reported some form of prior mental health care. While 89 (74.2%) had previously used an AI chatbot, only 30 (25.2%) had used one for mental health support or information.

This study was approved by Pearl IRB (Indianapolis, IN; Study# 2025‐0463), an independent institutional review board. All procedures were conducted in accordance with the ethical standards of the 1964 Helsinki Declaration and its later amendments. Consent was obtained electronically from all participants prior to participation. Usage was incentivized at US $50 per week for those who met the 60-minute threshold and US $30 for those who exceeded 30 minutes but did not reach 60 minutes. Including compensation for survey and interview completion, participants could earn up to US $275 in total. All quantitative and qualitative data were deidentified prior to analysis and stored on secure, access-restricted servers. The secure LLM (GPT-4o-mini) instance used for qualitative analysis did not retain participant data or use it for model training. Only authorized members of the research team had access to study data.

Open-ended written responses and transcribed video responses were analyzed using a structured, LLM-assisted qualitative content analysis approach designed to characterize recurring feedback patterns within qualitative data. LLM-assisted content analysis is an emergent qualitative technique that leverages LLMs’ strengths in pattern recognition and linguistic synthesis [38]. An inductive approach was appropriate given that our goal for this analysis was purely descriptive: that is, we aimed to characterize emergent themes rather than to apply prespecified theories or codes to the data [39]. The LLM was prompted to extract candidate themes related to user experience (both positive and negative) and to provide illustrative quotes directly drawn from participant responses. Themes were generated separately for each cohort to explore potential shifts as the chatbot prototype evolved across iterative refinements.

In keeping with qualitative rigor standards [40], several strategies were used to enhance trustworthiness. A reflexive approach was adopted to account for potential bias arising from the nonneutrality of the LLM and the researchers’ positionality as employees of the company developing the chatbot. To increase dependability, the analytic procedure was repeated independently 3 times using identical prompts, and theme labels and descriptions were compared across runs to assess stability and consistency in structure. To enhance credibility [40], the researchers reviewed LLM-generated themes and quotations to ensure that quotes were coherent, nonredundant, and accurately reflected the described pattern. To ensure confirmability, we retained only themes that were consistently reproduced across model runs and deemed coherent in relation to the dataset as a whole. Transferability was supported by a detailed description of the study context, participant characteristics, and prototype-stage setting, allowing readers to evaluate applicability to similar contexts.

The goal of this user experience pilot trial was to recruit a sufficiently large sample to characterize usability and acceptability, gather rich user feedback, and obtain preliminary estimates of symptom change to inform the design and justification of future full-scale trials; accordingly, it was not statistically powered to detect clinical effects. First, descriptive statistics were used to summarize responses to the first-impression survey and 1-week follow-up surveys, including items assessing perceived emotional safety, trust, sense of being understood, professionalism, personalization, acceptability, and perceived advantages relative to a human therapist. Mean item ratings and stacked bar distributions were calculated using all available data.

The PHQ-8 and GAD-7 were analyzed separately to capture domain-specific symptom change and also combined into a Patient Health Questionnaire Anxiety and Depression Scale (PHQ-ADS) composite representing overall symptom burden [26], as this is a common metric used in digital mental health studies [41,42]. The composite was included to facilitate interpretation of global symptom change rather than to introduce a distinct confirmatory endpoint. Consistent with the developmental aims of this pilot, all symptom outcomes were interpreted as exploratory indicators to inform future trial design.

Clinical outcome analyses were restricted to participants with at least mild symptoms at baseline for the given outcome. For depressive symptoms, models included only participants with PHQ-8 scores ≥5 at baseline; for anxiety symptoms, only those with baseline GAD-7 scores ≥5 were included. Models estimating total symptoms (PHQ-ADS) and secondary outcomes were estimated among participants who had either PHQ-8 or GAD-7 ≥5 at baseline (not necessarily both). Changes in clinical outcomes from baseline to the 2-week follow-up were estimated using linear mixed-effects models with a random intercept for participant, with time (follow-up vs baseline) as the main predictor and demographic covariates and cohort indicators included as fixed effects. Exploratory regression models tested whether first-impression ratings of the chatbot predicted 1-week therapeutic alliance and acceptability scores, adjusting for covariates and cohort. Finally, Pearson correlations examined associations between each first-impression rating and residualized symptom change scores (ie, calculated by regressing follow-up symptom scores on baseline severity and using the residuals as indices of symptom change independent of baseline levels), providing preliminary evidence for experiential predictors of improvement.

This study was reported in accordance with the CONSORT (Consolidated Standards of Reporting Trials) extension for pilot and feasibility trials (Checklist 1).

Consistent with the early-stage, developmental focus of this pilot, qualitative analyses were exploratory and used primarily to identify actionable areas for product refinement. Across the 3 cohorts, participants described the chatbot as accessible, easy to use, and nonjudgmental, but limited by repetitive and impersonal responses. Qualitative content analysis identified three primary domains of experience: (1) accessibility and usability, (2) emotional validation and reflection, and (3) limitations in personalization and conversational depth.

Participants across all cohorts described the chatbot as a convenient outlet for emotional expression and appreciated being able to engage at any time and fit sessions flexibly into their routines. As one participant explained, “The chatbot was able to express my feelings without any judgment, and it’s there whenever I needed to talk.” [Cohort 1]. Others highlighted the app’s intuitive interface and customizable sessions, calling it “super easy to use. To set up... super easy. It may be the simplest app I’ve ever used to actually get in and do that.” [Cohort 2] and appreciating that “it was handy. I could use it at any point in time 24/7.” [Cohort 3]. These qualities made the chatbot particularly appealing as a low-barrier, on-demand support tool.

Across cohorts, participants also valued the chatbot’s ability to validate emotions and encourage self-reflection. Many described feeling heard and supported when the chatbot acknowledged their emotions, with one noting that it “did a good job validating my feelings and helping me remember that others feel the same way” [Cohort 1]. Later users reported that it facilitated goal setting and simple coping strategies, such as reminders to take breaks or practice breathing exercises: “It helped me set a goal, did a grounding exercise, and gave me tools to get started” [Cohort 3]. These elements contributed to perceptions of the chatbot as supportive, though somewhat limited in depth.

Despite these strengths, the chatbot’s repetitiveness and lack of personalization emerged as key pain points. Participants frequently described conversations as “robotic” or “circular,” with limited responsiveness to context. One Cohort 1 user noted, “It just kept asking the same questions and using the same exact language,” while a Cohort 2 participant remarked, “Every response was starting with my name... it seems very unnatural and stiff.” Others lamented that the chatbot failed to pivot or offer specific advice: “It kept asking me the same questions... whereas a human could pivot the conversation” [Cohort 3]. Feedback on session structure also shifted across cohorts. Cohort 1 users requested more guidance around session length, which informed the introduction of timed sessions; however, Cohorts 2 and 3 reported occasional frustration when session cut-offs truncated ongoing discussion.

Across cohorts, core themes were consistent: participants valued accessibility and emotional validation but critiqued repetitive and impersonal responses. Cohort-level differences primarily reflected emphasis on emotional safety in Cohort 1, usability and goal-setting in Cohort 2, and coping tools and session design in Cohort 3. As the prototype evolved across cohorts, feedback appeared to shift from general impressions to more specific critiques of functionality and conversational adaptability. Overall, participants saw clear potential in the chatbot while emphasizing that improvements in personalization, conversational nuance, and responsiveness will be essential for enhancing its value.

Usability, assessed using an adapted Intervention Usability Scale (IUS) [27] specific to the mental wellness chatbot (0‐100 scale), was generally favorable. Participants reported a mean IUS score of 75.0 (SD 13.6). Although the IUS does not have established percentile norms, scores in the mid-70s are interpreted as “good” usability on the original System Usability Scale [43]. For context, the mean usability rating in this study was higher than that reported for the motivational interviewing intervention in the original IUS validation paper (68.7), suggesting that the chatbot app was, on the whole, experienced as usable by participants. Household income was not correlated with usability scores (r=0.07, P=.47).

Participants generally reported moderately positive first impressions of the chatbot’s emotional safety, trustworthiness, sense of being heard and understood, and professionalism, with mean ratings hovering slightly above the scale midpoint on most items (Figure 1A). However, a substantial minority of participants endorsed neutral or negative response options, particularly for items related to feeling deeply understood (≈40%) and perceived personalization of responses (≈65%), indicating focal opportunities for improvement. At the 1-week follow-up, perceptions of emotional safety, comfort, and professionalism were generally stable (Figure 1B), although ratings of response personalization declined relative to first impressions.

Acceptability ratings at one week reflected a similar pattern of mixed strengths and areas to improve upon (Figure 2A). Most participants agreed or strongly agreed that they trusted the chatbot to respond appropriately and understood how the app was intended to help. However, broader indicators of acceptability were notably lower. Fewer than 10% strongly agreed that they liked interacting with the app, and fewer than half agreed or strongly agreed overall. Perceived helpfulness was also modest, with only ≈40% agreeing that the chatbot’s responses were helpful for their mental health. Ratings of self-efficacy also highlight an area for improvement, with fewer than half agreeing that they were able to use the app consistently and correctly. Household income was not related to first impression ratings (r=0.05, P=.59) or overall acceptability (r=–0.09, P=.37).

Therapeutic alliance ratings at 1-week follow-up were moderate across bond, task agreement, and goal agreement subdimensions, with mean item scores just narrowly above the scale midpoint (Figure S3 in Multimedia Appendix 1 [26-37]). Compared with human therapists, participants generally viewed the chatbot as more accessible, less stigmatizing, and easier to open up to and be honest with (Figure 2B). Taken together, these findings indicate that while the chatbot was generally viewed as safe, respectful, and understandable, with some key advantages over traditional therapy settings, participants’ experiences of enjoyment, perceived helpfulness, confidence in use, and personalization were more variable and less positive, on average. Household income was not related to perceived therapeutic alliance (r=−0.11, P=.28).

Exploratory models examined which specific experiences, assessed at the 1-week follow-up, were associated with therapeutic alliance and acceptability (Table 4). Although the trust item was removed due to multicollinearity issues (ie, Variance Inflation Factor >4), several experiences emerged as relevant correlates. Participants who felt more emotionally safe, heard/validated, and understood by the chatbot reported significantly higher therapeutic alliance scores and greater overall acceptability. Additionally, perceived personalization of the chatbot was significantly related to therapeutic alliance and acceptability, though with a slightly smaller effect. Taken together, participants’ sense of emotional safety and validation from the chatbot were the strongest experiential correlates of therapeutic alliance and acceptability.

These models also revealed notable demographic patterns. Relative to White participants, those identifying as Black, Hispanic, or multiracial/another race reported significantly higher acceptability, and Black participants also reported higher alliance. Additionally, participants in Cohort 2 (and for alliance, Cohort 3) reported higher ratings than those in Cohort 1, possibly reflecting iterations made to the chatbot based on the first cohort’s feedback.

This mixed methods pilot evaluated (1) user experience and acceptability, (2) feasibility of safety monitoring, and (3) an exploratory clinical signal for an early-stage generative AI mental wellness chatbot. Under an incentivized, time-limited protocol, participants reported the chatbot as accessible and emotionally validating while noting limitations in personalization and conversational depth. Safety workflows were feasible: 8% of participants triggered an automated safety alert, all were reviewed within 2 hours, and none required outreach under prespecified criteria. Models showed small-to-moderate pre–post reductions in depressive and anxiety symptoms, with larger predicted improvements among participants with higher baseline severity, though model-predicted effects at the upper end should be interpreted cautiously given sparser data among those with elevated baseline symptoms. Together, findings support procedural feasibility and identify candidate therapeutic targets to inform ongoing refinement and future controlled trials.

Collectively, these mixed methods findings suggest that even a minimally refined generative AI–based chatbot can deliver a safe and accessible user experience that may be associated with modest short-term improvements in mental health symptoms under pilot conditions. The iterative cohort design highlights the value of rapid user feedback loops for refining conversational flow, session pacing, and safety monitoring protocols. As LLM-based systems continue to evolve, improvements in contextual coherence, personalization, and adaptive responsiveness may strengthen user experience and engagement, but these possibilities require evaluation under more rigorous, controlled conditions.

As chatbot systems are iteratively refined, tracking theory-consistent mediators, such as experiential and behavioral change processes, across product versions will be critical to determining whether design modifications meaningfully enhance user experience and clinical signal. Longer-term randomized studies under more naturalistic conditions are also needed to evaluate durability and to examine how chatbots may operate within integrated care pathways; for example, as a pretherapy intake tool, a between-session reinforcement, a maintenance tool following treatment, or a monitored entry point that escalates to human clinicians when symptom trajectories fail to improve or deteriorate [52]. Finally, future work should continue to explore the balance between usability and therapeutic rigor, as highly engaging conversational interfaces must also preserve core therapeutic processes that may not always prioritize immediate gratification.

Regarding limitations, the brief, 2-week exposure period limits conclusions about durability, particularly because outcomes were measured during or immediately after active use. Such designs may capture proximal engagement effects rather than sustained change [14]. However, this timeframe was appropriate for a pilot focused on usability, acceptability, and early clinical signals, and it enabled rapid prototype iteration. The single-arm pre–post design also raises the possibility of regression to the mean. In the absence of a randomized comparison group, improvements cannot be attributed definitively to the chatbot intervention, yet the magnitude of observed improvements—especially at higher baseline severity—appears larger than short-term changes typically reported in waitlist-controlled meta-analyses [53,54]

Participants were recruited from a convenience panel familiar with providing user-experience feedback on digital apps (dscout), which may limit generalizability, but provided high compliance and rich qualitative feedback appropriate for early-stage evaluation. Relatedly, pilot sampling purposefully excluded individuals with acute risk and those currently in treatment, consistent with the nascent state of this chatbot app. Engagement during this pilot was supported by usage-contingent incentives to ensure adequate exposure for prototype evaluation. This structure was appropriate for assessing usability, safety monitoring procedures, and early experiential processes, but engagement metrics should be interpreted as reflecting the feasibility of the study protocol rather than voluntary real-world uptake. While such incentives are common in early prototyping research, future effectiveness trials should evaluate voluntary, real-world engagement. Moreover, incentive structures may have influenced not only engagement patterns but also self-reported experiential and clinical outcomes, introducing potential response bias and limiting the internal validity of these self-reported measures. Although exploratory analyses did not detect significant associations between household income and self-reported experiential or clinical outcomes, income data were missing for 13.6% of participants, which may introduce selection bias if nonresponse was systematic.

Finally, the qualitative aspects relied on LLM-assisted content analysis to identify recurring experiential patterns rather than on researcher-led reflexive thematic analysis. Although theme stability was assessed across multiple independent model runs and outputs were reviewed for coherence, this approach did not incorporate traditional qualitative features such as prolonged coding immersion, formal reflexivity procedures, or independent human coding. Accordingly, qualitative findings should be interpreted as descriptive and exploratory, appropriate to the formative aims of this pilot study.

This pilot user experience study establishes critical feasibility and proof of concept for an early-stage mental well-being chatbot that can deliver meaningful and safe usability and measurable short-term symptom improvement, even in a mildly symptomatic sample. Transparent documentation of iterative development, combined with mixed methods insights and severity-based marginal effects, highlights both promising preliminary findings and the design targets most likely to enhance future engagement and clinical impact. These findings guide continued refinement and set the stage for rigorous, longer-term trials evaluating sustained engagement, clinical durability, and performance in more symptomatic samples.

Beyond this prototype, the study demonstrates a practical evaluation template for early-stage LLM mental health tools that combines user experience, safety workflow feasibility, and exploratory clinical outcomes with explicit caution about efficacy inference. As mental wellness chatbots continue to develop, transparent reporting of design constraints, safety monitoring, and user-perceived therapeutic processes will be critical for responsible iteration and for interpreting early clinical signals prior to larger randomized trials and real-world deployment.