The MI Dialogue Engine contains the core logic that translates MI principles and strategies into a natural language capable CA. The engine can be configured to use any off-the-shelf foundation models, for example, OpenAI’s GPT-4.5 [33] or Meta’s Llama 3 [34]. To generate coherent and therapeutically aligned dialogue, we augment the foundational model using a combination of techniques such as workflow-routing and prompt-chaining [35].

Specifically, the Dialogue Engine is composed of an orchestrator that routes every message through the workflow described in Figure 1. A node on the workflow is an atomic task that is carefully mapped to an MI concept and consists of a prompt and an invocation to the LLM. For instance, the Off-topic Detection task invokes the LLM with a prompt instructing the model to detect off-topic client utterances. The orchestrator then evaluates a condition (or a set of conditions) on the output to route the message to the next appropriate task in the workflow. In this way, outputs are chained together to produce safe and MI-consistent responses.

To effectively emulate an MI session, the workflow is designed to guide the conversation along the four stages of MI, that is, engaging, focusing, evoking, and planning. This is achieved sequentially in the following three phases: (1) identifying whether the goals within the current MI stage are met and advancing the stage if appropriate, (2) applying stage-specific MI strategies to draft a response strategy, and (3) crafting the actual response. The tasks within each phase are as follows:

The principal objective in this phase is to determine the current MI stage. At the beginning of each session, the MI stage is set to engaging. As the conversation advances, the Stage Analyzer task ascertains if the goals associated with the current MI stage have been achieved. These goals, derived from the MI handbook [9], are provided to the task as summarized JSON strings. Once the goals are met, the orchestrator advances the MI stage sequentially (engaging → focusing → evoking → planning). The resulting MI stage is subsequently used by downstream tasks in Phases 2 and 3.

Simultaneously, 2 other auxiliary tasks also monitor client utterances to ensure the conversation remains aligned to the broader intervention objectives. The off-topic detection task identifies if the client’s message potentially diverges from the focus—for instance, expressing job stress or work dissatisfaction in a session aimed at improved physical activity. If detected, a special off-topic strategy is applied in the subsequent phase; the client’s message is acknowledged, but an attempt is made to gently steer back to the primary topic. On the other hand, the Refocus Detection task identifies if a client appears to redefine their goals in the planning stage of MI. For example, expressing doubt about completing an existing goal and indicating interest in taking a different direction. In this case, the MI stage is rolled back to evoking, and an attempt is made to reinforce commitment and planning toward the new goal.

Once the MI stage has been reliably detected, the workflow enters the Response Planning phase. Here, a context-appropriate response plan is generated based on (1) the current MI stage, (2) the intervention specification, and (3) the conversation history with the client. It consists of a single task that provides the LLM with a list of MI strategies appropriate for the current MI stage. For instance, if the current stage is Engaging, the system selects stage-appropriate techniques such as OARS (along with other stage-specific techniques) to generate an appropriate response plan.

Finally, the generated plan is consumed by the Response Generation task to finally produce the actual response. This pattern—where the response planning is decomposed into an intermediate step—has been shown to produce higher quality outputs and enhance adherence to MI principles [26]. The MI Dialogue Engine passes this generated response to the Bot Gateway, which eventually delivers it to the client. All prompts used to guide the model are provided in Multimedia Appendix 2.

We conducted a foundational study to assess the performance of Aimi, in terms of its fidelity to MI counseling (RQ1) and users’ perceived interaction quality (RQ2), when compared against human and rule-based coaching approaches. We focused on physical activity as a controlled test case.

We conducted a mixed methods human evaluation study between February and March 2025 using a within-participants design to explore the performance of the following three types of MI coaching approaches:

We recruited a convenience sample of healthy adults aged 21‐59 years who were fluent in English (Table 1). Participants were located in Singapore and were recruited through study advertisements on the research center’s internal chat forum and mailing lists.

Participants attended a 1.5-hour in-person, laboratory-based study session. After receiving a verbal explanation of the study procedures and providing written informed consent, each participant was provided with a laptop and an instructional leaflet. The leaflet outlined (1) how to access the coaching sessions, (2) the order in which to complete them, and (3) the steps for submitting postsession feedback. Each session was assigned a unique identifier to link the conversation transcript with the corresponding postsession survey. Due to logistical constraints and longer response times associated with the novice human coach, true probabilistic randomization was not feasible. Instead, a quasi-randomized approach was used whereby participants completed the 3 coaching sessions in a predefined sequence dictated by operational availability. While the specific sequence allocation was blinded to the participants, the conditions were not fully counterbalanced across all possible orderings.

Participants were instructed to assume the role of a physically inactive individual to provide a consistent baseline across the approaches. When ready, participants activated a coaching session through a web-based chat interface accessed via a laptop, using a unique ID that was linked to the allocated condition for randomization and tracking. They then engaged in a text-based conversation with one of the three MI coaches—Aimi, a human coach, or a rule-based coach—without being informed which coach they were interacting. However, even though blinding was attempted, it could not be strictly enforced because of the inherent differences in the technical characteristics of the conditions (eg, response latencies, predefined responses, and so on). Each coaching session lasted approximately 20 minutes, depending on the conversational flow. At the end of the interaction, participants completed a short survey evaluating (1) working alliance with the coach, (2) dialogue relevance, (3) linguistic quality, (4) empathy, (5) engagement, (6) usability, (7) motivation to change, and (8) open-ended feedback on the experience. This process was repeated until participants had completed all 3 coaching sessions and associated evaluations.

At the end of the study, each participant was reimbursed with an e-voucher valued at SGD 30 (approximately US $23).

Upon first completing the short survey, participants were asked to report demographic characteristics, including gender, age, nationality, and education level, as well as their physical activity level and prior experience using chatbots.

The quality and fidelity to MI were assessed by an expert member of the Motivational Interviewing Network of Trainers using the Manual for the Motivational Interviewing Skill Code Version 2.0 (MISC-2) [37]. Specifically, each conversational turn in the transcripts was coded in relation to the behavior goal. The coding system produced the following three scores: Behavior Counts, Summary Scores, and Global Ratings.

Behavior Counts capture specific coach and client behaviors through sequential coding, independent of global ratings. There are 15 major categories of coach behavior in MISC-2: advise with and without permission, affirm, confront, direct, emphasize control, facilitate, filler, giving information, open and closed questions, raise concern with and without permission, simple and complex reflection, reframe, support, structure, and warn. Counts are determined through coding and tallying the frequency of specific language behaviors using 19 coach codes and 7 client codes. Client statements are attributed either a positive or negative valence that reflects language (ie, reason, desire, ability, need, commitment, and taking steps) toward or away from the behavior goal (change or sustain talk).

Summary Scores are calculated from behavior count frequencies to provide concise indicators of MI competence and session dynamics. These include ratios (eg, reflections-to-questions ratio and open-to-closed questions ratio) and percentages (eg, percent complex reflections and percent MI-consistent behaviors) that quantify the coach’s adherence to MI principles and the client’s engagement in change language.

Global Ratings are based on an overall holistic impression of the entire coach-client interaction. Ratings consider three global coach dimensions—acceptance (unconditional positive regard), empathy (understanding the client’s perspective), and MI spirit (encompassing collaboration, evocation, and autonomy support)—and 1 client dimension (self-exploration). All ratings use a 7-point Likert scale (1-7), with coders beginning at a baseline score of 4 and adjusting upward or downward based on the coach’s or client’s performance throughout the session. Higher scores indicate higher quality.

In the absence of a validated evaluation tool for human-LLM interactions, we used questionnaire items from previous research studies evaluating CAs [38]. We list the items below and provide the full questionnaire in Multimedia Appendix 3.

Session alliance was measured on a 6-point Likert scale anchored from 1 (“not at all”) to 6 (“completely”), while the remaining outcomes were measured on a 5-point Likert scale anchored from 1 (“strongly disagree”) to 5 (“strongly agree”).

Response latency was defined as the elapsed time between a user input and the corresponding coach response and was computed for all sessions. Latency distributions were summarized using median, mean, SD, and percentile-based thresholds.

Participants provided contextual insights into the strengths and weaknesses of each coaching interaction, which are summarized in Table 4.

Participants described Aimi as demonstrating contextual understanding and adaptive responsiveness, creating a fluid conversational experience that was relevant to their inputs. They appreciated the coach’s ability to maintain coherence even when conversations diverged from the initial direction. One participant noted:

Participants also highlighted that the sessions were personalized:

The conversational flexibility fostered engagement, with one participant describing it as

However, this conversational strength was undermined by excessive reflection and summarization. Participants consistently described this feature as robotic and frustrating, with the most frequent criticism being that the coach’s paraphrasing felt redundant. One participant explained:

Another elaborated:

Despite Aimi’s ability to elicit motivation, the excessive questioning without concrete guidance limited its practical value. Participants were frustrated by the high question-to-suggestion ratio and the coach’s reluctance to provide direct advice. As one user described it:

Another participant expressed similar frustration:

The novice human coach was perceived to be more emotionally attuned, and participants could sense that responses were tailored to their needs based on genuine human understanding. Participants consistently emphasized the authentic quality of the interaction and the coach’s commitment to exploring deeper meaning and individual context. One participant noted:

The human touch was also widely appreciated, with one participant describing the experience as

However, despite these relational strengths, slow response times significantly compromised the user experience. The latency problem was so severe that it offset the conversational advantages and created user frustration. One participant captured this tension:

Another stated bluntly:

Beyond the responsiveness issue, participants also noted that emotional support did not translate into tangible planning or concrete guidance. While the human coach provided some general tips, participants felt the coaching stopped short of actionable next steps. One user reflected:

but another expressed the limitation:

The rule-based CA delivered rapid, structured interactions with clear clickable response options that many users appreciated for convenience. Participants valued the streamlined experience:

and

The speed and simplicity were genuine strengths for users seeking quick information delivery.

However, this efficiency came at the cost of conversational depth and relational engagement. Participants consistently described the interaction as overly factual, constrained, and emotionally disengaged. The predefined answer options, while convenient, limited the direction and authenticity of the conversation. One participant explained:

while another emphasized the need for flexibility:

The constraining nature of the interaction was particularly noted:

Despite providing informative content, participants felt that the coach did not help translate this information into personalized action plans. One user reflected:

Another noted:

Additionally, a common criticism was that the coach’s predefined responses often felt assumptive and reduced trust:

Consistent with recent literature, our findings reinforce the growing evidence that artificial intelligence (AI)–driven MI coaching is feasible to implement and may enhance users’ motivation to change health behaviors. In a recent scoping review, Karve et al [41] summarized studies using AI for MI delivery, reporting that most interventions, whether rule-based or AI-based, demonstrated improvements in motivation, confidence, and readiness to change across domains such as physical activity [26,38], smoking cessation [23-25], alcohol and substance use [21,22,30,42], and general lifestyle [27,28,43]. Our study extends this literature by contextualizing the performance of a structured LLM-workflow–based CA alongside a novice MI-trained human coach and a rule-based CA, a gap not explicitly addressed in prior work.

Our findings show that participants rated Aimi as broadly equivalent to a novice MI-trained human coach across measures such as linguistic quality, dialogue relevance, and engagement, in line with recent reports that digital coaches and AI agents can provide relational support [44-46]. However, Aimi outperformed the human coach in terms of response latency (median 4.80, IQR 4.3-5.4 seconds vs median 38.09, IQR 18.1-75.73 seconds), with participants frequently reporting disengagement due to long delays in human responses. This observation underscores a key advantage of CAs in delivering scalable, on-demand intervention support unconstrained by human resource limitations and constrained primarily by the underlying LLM’s inference throughout, which is already capable of serving billions of prompts daily [47].

In terms of MI fidelity, Aimi achieved a median of 100% MI-consistent utterances. This aligns with other LLM-based MI systems that show MI consistency can approach, or even slightly surpass, human performance. For instance, Jörke et al [26] reported 93.3% MI-consistent utterances in a physical activity intervention, while Yang et al [43] found 96.6% MI-consistency using GPT-4o as an underlying LLM for transtheoretical behavioral change coaching. Furthermore, Aimi could also consistently evoke client change-talk. Given that eliciting change-talk is the core theoretical mechanism of MI, these results suggest that Aimi is capable of operationalizing MI.

However, such high fidelity scores also uncover an interesting “fidelity-engagement” paradox. Despite high expert-rated technical scores, feedback from participants suggested that Aimi’s frequent usage of reflections and summarizations felt excessive or robotic, which negatively affected their experience. Similar findings were reported by Steenstra et al [22], where the agent’s overreliance on complex reflections was perceived as redundant. Rather than running in tandem, our results highlight an important trade-off between therapeutic fidelity and authentic engagement. Furthermore, an overuse of reflections might also trigger the so-called “uncanny valley,” where excessive expressions of empathy appear performative rather than genuine [48]. Ultimately, this suggests that future LLM-based CAs must learn to balance technical accuracy with natural conversation rather than solely optimizing for strict protocol adherence.

Prompt design is increasingly recognized as a key strategy to improve the structure of coaching sessions in LLM-based MI delivery. Prior studies [22,24,27,30] have typically adopted single-prompt approaches, in which the LLM is instructed to act as an MI expert by specifying the desired persona, tone, and conversational structure. While straightforward, this approach is prone to model hallucinations and conversational drift as the session progresses [26,31]. In contrast, our study adopted a workflow-based prompt-chaining strategy, which decomposes lengthy and complex instructions into smaller, manageable tasks aligned with specific MI dialogue states. This method provides more reliable guidance and reduces the risk of drift throughout the conversation.

The workflow approach may also explain the system’s technical performance in eliciting change talk. By systematically monitoring progression through strict MI stage-based goal verification, the Stage Analyzer task ensured that Aimi refrained from offering advice too early or prematurely rushed to the planning stage. Ultimately, by structurally enforcing adherence to MI principles, the system potentially mirrored the behavior of a highly disciplined MI practitioner. Comparable strategies have emerged in the literature. For example, Jörke et al [26] implemented a prompt chain to guide physical activity coaching sessions across defined “dialogue states” (eg, motivation and goal setting), applying MI strategies conditioned on each state. Similarly, Meyer and Elsweiler [28] combined template-driven framing with natural language understanding for enhanced valence classification, generating free-text responses only during selected MI phases (eg, evoking). Together, these workflow-based and hybrid methods represent best current practices to balance flexibility, safety, and conversational coherence in LLM-mediated MI delivery.

A recurring theme from participant feedback was the expectation that the coach would provide immediate actionable advice. However, MI requires self-exploration, which is inherently effortful, requiring sustained cognitive engagement from users. User adoption and continued use depend heavily on perceived value and usability, and interactions that feel too effortful or misaligned with user expectations may lead to disengagement [49,50]. Conveying the purpose and demands of MI-oriented interactions may be a critical step when implementing digitally-delivered MI to ensure user expectations are aligned with MI and to prevent disengagement.

Within MI, effective behavior change also depends on the relational capacity of the coach. Specifically, MI theory distinguishes between two active ingredients that jointly facilitate behavior change: the technical component, encompassing specific counseling strategies and processes (eg, reflections, open questions, and elicitation of change talk), and the relational component—often referred to as the “spirit” of MI—reflecting empathy, collaboration, and the quality of the therapeutic relationship [9]. In traditional in-person MI sessions, these components are tightly interlinked. Skilled practitioners draw on nonverbal and paralinguistic cues, such as tone of voice, facial expressions, and body language, to balance technical delivery with relational responsiveness, tune in to emerging change talk, and adapt the conversation in real time. However, in a purely text-based format, the lack of access to such cues constrains the CA’s capacity to enact the more relational dimension of MI delivery. This limitation may help explain why some participants perceived Aimi as less useful or responsive, especially those seeking more immediate advice.

On the other hand, the relative strength of digital coaching agents may lie precisely in the technical aspects of MI. By applying MI-consistent strategies, LLM-based CAs may support users in clarifying their motivations and readiness for change, even in the absence of a fully realized relational component. From this perspective, MI CAs may function less as substitutes for human counseling but more as adjunctive or preparatory tools, providing an initial impetus for reflection that encourages users to seek further in-person support where the relational ingredient of behavior change can be more fully realized. Future digital coaching systems may therefore benefit from explicitly positioning LLM-based MI CAs as complementary supports that prioritize technical fidelity while facilitating onward engagement with blended human-delivered interventions.

Additionally, the effectiveness of this technical delivery is likely to be contingent on users’ readiness to engage in reflective change processes. Individuals in earlier stages of change (precontemplation or contemplation) may be more receptive to reflection-based MI strategies that build awareness and intrinsic motivation, whereas those in later stages (preparation, action, or maintenance) may benefit from directive MI-consistent techniques, such as goal setting, action planning, and strengthening commitment to change [51]. This suggests that LLM-based MI CAs should not be conceptualized as static implementations of MI, but as adaptive systems capable of modulating technical strategy in response to users’ stage of change. Such tailoring aligns with evidence from mobile health and digital MI research indicating that user engagement and satisfaction improve when interaction style is matched to readiness and individual needs [52].

Furthermore, this adaptive approach can be also extended to alleviate the “intention-action gap”—while MI is designed to strengthen a client’s intrinsic motivation and intention to change, the CA cannot inherently ensure real-world action. To bridge this, future LLM-based CAs could integrate with objective wearable data to monitor behavioral outcomes [53]. If the system detects that a user did not execute a planned behavior (eg, a scheduled walk), the CA could dynamically adjust its strategy, shifting from action-oriented planning back toward exploring ambivalence or rolling with resistance.

Finally, users in this study appreciated the speed, structure, and low-effort interaction of the rule-based coach but perceived it as too rigid, generic, and impersonal to support behavior change effectively. This highlights that rule-based and LLM-based approaches each offer distinct yet potentially complementary strengths in the delivery of MI. A promising future direction may lie in developing hybrid architectures [44,54,55] that leverage the reliability and efficiency of rule-based structures alongside the adaptive, context-aware capabilities of LLMs.