The Consolidated Criteria for Reporting Qualitative Research checklist was used in the design and reporting of this study [30]. Semistructured, one-to-one interviews were conducted.

A constructivist theoretical framework was used to inform data collection and analysis [31]. This theoretical approach was selected because constructivism recognizes the active role of the researcher in the generation and interpretation of qualitative data.

For pragmatic purposes, participants were recruited across 2 periods: June 2020 to August 2020 (led by KS and OP) and April 2021 to August 2021 (led by AA and OP). Owing to the COVID-19 pandemic, it was challenging to recruit as planned during the summer of 2020. Therefore, we continued the recruitment in 2021. The project team decided that it would be useful to recruit participants from 2 different subgroups (ie, past-year smokers who had voluntarily used Quit Coach in a recent smoking cessation attempt and current smokers who agreed to download and use Quit Coach for a minimum of 2 weeks to support a new cessation attempt) as a form of triangulation [32]. We reasoned that such triangulation of results when varying the eligibility criteria (rather than the methods) would either help to validate the results (eg, if smokers who did not self-select to download and use the Smoke Free app had similar experiences with Quit Coach as those who had voluntarily used the app) or highlight different experiences owing to smoking status or treatment-seeking behavior.

Participants recruited in 2020 were eligible to participate if they (1) were aged ≥18 years, (2) were fluent English speakers based in the United Kingdom, (3) were past-year smokers and had used the pro (ie, paid) version of the Smoke Free app (ie, the app version that included Quit Coach) for at least two weeks, and (4) had interacted with Quit Coach at least once during the 2-week period.

Participants recruited in 2021 were eligible to participate if they (1) were aged ≥18 years; (2) were fluent or highly competent English speakers, with no restrictions on geography; (3) were current cigarette smokers; (4) were willing to make a quit attempt within 1 week from initial contact with the researchers and use Quit Coach for at least two weeks; and (5) owned a smartphone.

All the participants used the pro version of the Smoke Free app for a period of at least 2 weeks before participating in the semistructured interviews. We expected this time window to be sufficient for enabling detailed conversation about participants’ chatbot experiences.

Participants recruited in 2020 were approached through advertisements (unpaid) shared on social media platforms (ie, Facebook and Twitter) and through a mailing list of Smoke Free app users. The recruitment materials stated that Smoke Free users were invited to participate in a web-based interview about their experiences with the app (good or bad), with particular focus on the Quit Coach feature. Participants were incentivized to win 1 of 5 gift vouchers worth £20 (approximately US $24).

Participants recruited in 2021 were approached through advertisements (unpaid) shared on social media platforms (ie, LinkedIn, Facebook, and Instagram), directly through the researchers’ networks (ie, WhatsApp, email, SMS text messages, and flyers), and through professional web-based recruitment platforms (ie, Prolific and Call for Participants). The recruitment materials stated that smokers interested in making a quit attempt with the use of a smartphone app were invited to participate in a web-based interview about their experiences with the app (good or bad), with particular focus on its chatbot feature. Participants received a gift voucher worth £10 (approximately US $12) after completing the interview.

Participants were recruited in batches of 4 to 5 participants each until theoretical saturation was judged to have occurred (ie, a point in the data collection process when no new information alters the identified themes) [33]. Preliminary data analysis was conducted by KS and subsequently by AA after each batch of 4 to 5 participants, to determine whether additional participants were needed.

Upon expressing interest, participants were asked to read the participant information sheet, provide informed consent, and complete the eligibility questionnaire via Qualtrics (Qualtrics International Inc). If eligible, participants recruited in 2020 were contacted by the researchers to arrange the interview. If eligible, participants recruited in 2021 were contacted to confirm study acceptance and provided with information on how to download the pro version of the Smoke Free app (using a free access code). Participants were asked to select a quit date and nominate a day that was 2 weeks from their quit date to complete the interview.

Owing to the COVID-19 pandemic, in-person interviews were not possible. Therefore, interviews were conducted via the web by KS or AA (graduate students enrolled in an MSc program in Behavior Change) via Microsoft Teams. Besides the participant and the researcher, no one else was present during the interviews. KS had limited experience in conducting qualitative interviews before this study; AA had extensive experience from working for a consultancy firm. Before conducting the interviews, OP (PhD in Health Psychology, extensive experience in conducting qualitative interviews through previous academic work) provided training to KS and AA. Interviews were audio recorded and lasted between 30 and 45 minutes. Following completion, the participants were thanked, verbally debriefed, and presented with an incentive.

Smoke Free [35] is an evidence-informed app with a large user base (approximately 4000 downloads per day). The app contains behavior change techniques that are expected from theory and evidence from other settings to aid smoking cessation [17,36]. Refer to the study by Jackson et al [37] for a summary of the behavior change techniques included in the Smoke Free app, coded against a 44-item taxonomy of techniques used in individual behavioral support for smoking cessation [36].

The pro (ie, paid) version of Smoke Free contains a text-based, quick-response chatbot called Quit Coach (Figure 1). During the first 2 weeks of a user’s quit attempt, Quit Coach initiates twice-daily check-ins with users regarding their cessation attempt through a push notification. Check-in frequency is programmed to reduce after 1 month and cease entirely after 90 days (when users are anticipated to have quit smoking). Users engage with Quit Coach via text messages, selecting from prewritten responses. There are a few exceptions to this format, such as when users complete certain exercises that require free-text input (eg, typing the mantra, “not another puff, no matter what”), to occasionally type what is influencing their craving, or for providing feedback on whether they found a piece of advice useful (refer to Multimedia Appendix 1 for additional screenshots of such interactions). A bespoke Node .js natural language processing framework (adapted from freely available, state-of-the-art source code by Smoke Free’s developers) is used to map free-text inputs onto their likely intent—this constitutes the only machine learning element of the chatbot. Users can also initiate check-ins themselves by opening Quit Coach to record a craving or ask for assistance via a get help now toolkit, which provides different options for directing the conversation with Quit Coach. The conversational options include craving management, relapse, difficult situations, and withdrawal. Quit Coach’s communications (typically in text form, but also through emojis and Graphics Interchange Formats [GIFs]) contain information about the health consequences of smoking, quitting tips, and motivational messages, simulating a text message conversation.

Interviews conducted in 2020 and 2021 were combined to form a single data set. Analysis was performed through an inductive thematic approach following the methodology by Braun and Clarke [38,39]: (1) familiarizing with the data, (2) generating initial codes, (3) searching for themes, (4) reviewing the themes, (5) defining and naming the themes, and (6) producing the report. We focused on latent (rather than semantic) meanings [39]. Although the Model of Supportive Accountability was used to inform the interview topic guide, terminology from this model was used for coding only if deemed relevant, with alternative codes considered throughout the process.

Interviews were coded using Microsoft Word in batches of 4 to 5 by AA to facilitate an iterative and reflexive approach. Each transcript was read multiple times to familiarize with the data. Then, initial codes were generated. Coded transcripts were reread following initial code generation and discussed with OP, adding or refining codes as appropriate. Then, the coded extracts were examined and used to generate preliminary themes. Next, a second, independent coder (another student from the same MSc program in Behavior Change) helped to assess coding reliability. The second coder coded 2 interviews, 1 each from the 2020 and 2021 samples, which were selected using a random number generator. The second coder was instructed to inductively code each interview. The resulting codes were compared with those of the first coder conceptually, rather than for perfect word matching. Discrepancies were discussed and reconciled. Then, themes were reviewed, refined, named, and agreed upon through discussion among AA, OP, and JB. During coding, the possibility for differences between the 2020 and 2021 samples was considered. Theoretical saturation was judged to have been reached after 12 interviews.

A subsample of 14% (2/14) of randomly selected participants were contacted and agreed to read the results and comment on the congruence of the themes and narrative generated by the researchers with their own experiences. Both participants agreed with the researchers’ interpretations.

The interviewers (women, White ethnicity, nonsmokers, and unfamiliar with most participants before the interview) felt that a good rapport was built with all the participants. Some were more immediately verbose, whereas others took a little time to open up, but did so with prompting and encouragement. For the first few interviews conducted, the interviewers closely followed the topic guide; however, as salient conversation topics emerged, a more discursive style of questioning was adopted to explore salient topics in great depth. Before commencing the interviews, participants were told about the goals of the study and that the interviewers were not directly involved in the development of the Smoke Free app; however, participants were unaware of the interviewers’ smoking status or theoretical assumptions regarding user engagement or smoking cessation.

Ethics approval for this study was obtained from University College London’s Research Ethics Committee (CEHP/2020/579). Participants provided written informed consent before participating in the study.

A total of 40 participants completed the screening survey and were eligible to participate in the study. Of the 40 participants, 26 (65%) participants did not complete an interview, as they later decided that it was not the right time to quit, became uncontactable, or failed to attend the interview. Table 1 shows a summary of the demographic and smoking characteristics of the 35% (14/40) included participants.

Using a qualitative approach, this study aimed to explore smokers’ experiences and engagement with a quick-response chatbot implemented within a popular smoking cessation app. Users’ experiences with the chatbot were largely positive. Anthropomorphism of the chatbot (ie, ascribing human-like characteristics, thoughts, and behavior to the chatbot) was enabled by its specific interaction style and format (eg, positive message tone and quick and easy-to-complete check-ins) and users’ perceived need for support, which appeared to give rise to feelings of accountability to the chatbot and increased engagement. Our results build on and extend previous qualitative findings pertaining to users’ experiences with companion chatbots [20,29] to a chatbot specifically designed to support smoking cessation.

A previous experimental study has shown that social responses to computers—that is, a direct consequence of anthropomorphism—are common and relatively easy to generate (eg, by providing the computer with human-like attributes, such as a language output) [40]. However, according to the uncanny valley hypothesis, chatbot designers have a fine line to tread [41]. Strong feelings of affinity are generated by great human-like qualities in a robot or computer program up to a point. Once it becomes very similar to or indistinguishable from a real human, people’s reactions can reverse because they may find them creepy or eerie [41]. Although the quick-response Quit Coach in this study was relatively simple (eg, it did not allow many free-text inputs from users) and made it clear to users that it is an automated bot, the presence of social cues (eg, its positive message tone and communication format similar to text messaging with GIFs and emojis) appeared to be sufficient for generating social responses from many users without entering the territory of the uncanny valley. This is positive, as it implies that simple (and relatively low-cost) chatbots generate feelings of affinity and that more complex chatbots that can more closely mimic human interactions may not be necessary.

However, users mentioned that they would have liked it if Quit Coach tailored its questions and responses to their unique situations and momentary needs. Therefore, future studies would benefit from exploring—for example, through user-centered design activities and experimental studies—additional design elements that can enhance Quit Coach’s similarity to humans, as this may further promote user engagement. In addition, some design elements appeared to detract from users’ anthropomorphism of and engagement with Quit Coach, such as its repetitive questions and responses and forced-choice interactions. For users who struggled to stay quit, the repetitiveness and inflexibility of Quit Coach appeared particularly salient, sometimes leading to a reversal of the anthropomorphic experience. Previous studies indicate that users’ perceived need for support and the target behavior itself (eg, progress toward smoking cessation) are important for continued engagement [9,10,15]. Similarly, high perceived need for support may be important for users to suspend disbelief and anthropomorphize conversational agents within the health care domain. The Three-Factor Theory of Anthropomorphism predicts that people are more likely to anthropomorphize nonhuman agents or objects (1) when anthropocentric knowledge is readily accessible and applicable (ie, when knowledge about how humans interact, think, and feel is judged as relevant for the interaction), (2) when motivated to be effective social agents (ie, motivation to master one’s environment by increasing its predictability and controllability), and (3) when lacking a sense of social connection to other humans (ie, feeling lonely or isolated) [42]. Future studies would benefit from building on and empirically testing such a theory of anthropomorphism within the human-computer interaction domain, with a view to improving the design of future conversational and relational agents for health and well-being.

Our findings also lend partial support to the Model of Supportive Accountability [22] in that users reported feeling accountable to checking in and updating the nonjudgmental and supportive Quit Coach. Finding a balance between nonjudgmental tone of voice and human-like social cues to generate feelings of affinity and accountability (as discussed previously) may be important for future behavior change chatbots. However, trust and competence (which are additional cornerstones of the Model of Supportive Accountability) were largely missing from users’ accounts. It is plausible that competence (eg, legitimacy of the information provided) and trust (eg, data security and confidentiality) were already assumed by users who had either voluntarily downloaded the Smoke Free app from a digital marketplace—likely selecting an app they trusted among the myriad of available apps [15]—or were asked to download it based on recommendation from university researchers. Alternatively, trust and competence may not be necessary conditions for supportive accountability to arise within human-chatbot relationships; this should be further explored in future studies.

This study was strengthened by recruiting both experienced and novice app users, having a second coder to help in validating the coding, using external validation to ensure that the researchers’ interpretations aligned with participants’ narratives, and achieving theoretical saturation. However, this study also had several limitations. First, self-selection bias may limit the applicability of the findings to other populations and settings. For example, the 36% (5/14) of participants recruited in 2020 had quit smoking successfully and were considered as heavy Quit Coach users, and most participants were young (ie, aged 18-44 years). Second, we did not record any additional support used by participants during their quit attempts (eg, pharmacological support), which may have influenced their perceived need for support. Third, for pragmatic purposes, we did not record participants’ actual engagement with Quit Coach, but instead relied on self-reports. Going forward, triangulation of qualitative and quantitative findings would be an important addition to the research literature. Fourth, the study was conducted during the COVID-19 pandemic—a time of significant change in people’s life and work conditions, including their smoking behavior [43]—which may also limit the applicability of our findings to other periods and contexts.

Findings from this study have both theoretical and practical implications. First, our results indicate that the Model of Supportive Accountability [22] may usefully be extended from human to human-like support within digital interventions. However, future studies should further explore the specific conditions under which chatbot interactions lead to feelings of accountability and whether accountability is more easily generated within human-to-human (rather than human-to-bot) interactions. Second, our findings suggest that chatbots for smoking cessation may benefit from including more variation in conversations to prevent boredom and incorporating different levels of tailoring (including context-sensitive tailoring).

Anthropomorphism of a quick-response chatbot implemented within a popular smoking cessation app appeared to be enabled by its interaction style and format (eg, positive message tone and quick check-ins) and users’ perceived need for support, which may have given rise to feelings of accountability and increased engagement.