Chatbots are used as a promising tool to promote COVID-19 vaccinations [1,2] as they offer the possibility of upscaled interactions with users. Chatbots can also support health communication by engaging users via social media channels [3]. Unfortunately, there is a considerable amount of resistance toward COVID-19 vaccinations in society [4,5], and chatbots themselves may evoke negative user responses (eg, [6]). Specifically, the effects of chatbot empathy have been found to be ambiguous. On the one hand, empathy displayed by a chatbot (ie, a chatbot showing its understanding of and care for the user’s experience) may lead to a positive user response [7]. It may also enhance autonomy support (ie, provide the user with a sense of freedom in acting upon the information received from the chatbot), which is considered to diminish psychological reactance [8,9]. On the other hand, previous studies [6,10,11] suggest that the consequences of artificial agents’ empathy may also be negative. These studies focus on the way of expressing empathy, suggesting that being too “humanlike” is harmful. However, the role of the context related to the conversation, as conversation quality and chatbot expertise cues, remains understudied. Since that conversation-related context is manageable, this gap is also important from the practical perspective of vaccination chatbots. Therefore, this study aims to bridge this gap by investigating how conversation quality and chatbot expertise cues moderate the effect of expressing empathy/autonomy support using a COVID-19 vaccination chatbot. We propose that the effect of expressing empathy/autonomy support is less positive (or more negative) when the actual answers provided by the chatbots fail to dissolve queries submitted by users, because users may have higher expectations for the quality of interaction if a chatbot is interacting in a more human way [12,13]. Given the technical complexity of chatbot development, especially in the context of topics that require a lot of domain-specific and rapidly changing knowledge, such as the COVID-19 vaccines, conversation fallback (CF, ie, the situation in which a chatbot is unable to answer a user’s question) is plausible [14,15]. In addition, users may react more positively to the chatbot empathy/autonomy support expression when they perceive the chatbot to be more trustworthy and competent [16], which is relevant to a controversial topic, such as vaccinations. Therefore, we considered an expertise cue in the form of the chatbot endorser's credibility as an additional moderator of the empathy/autonomy support expression effect. We examined these moderating effects in an experimental study with different levels of expression of empathy/autonomy support and endorser credibility for a COVID-19 vaccination chatbot (VaccinChat), considering the actual level of conversation quality.

According to the Behavioral Intervention Technology Model [17], our chatbot (VaccinChat) can be considered a behavioral intervention technology whose clinical aim is to increase users’ intention to get vaccinated against COVID-19. The chatbot patronage intention (CPI, including the willingness to use and recommend the chatbot) is considered the usage aim. The behavioral intervention strategy is education (ie, providing information about how vaccines work and how a user can get vaccinated). Accordingly, the chatbot endorser credibility (expert vs layperson) can be considered a relevant expertise cue contributing to the user's initial trust toward the chatbot as an educational tool. Behavioral intervention elements are the pieces of vaccine-related information provided by the chatbot. They can be characterized as having a textual form, being instantly delivered online, and being personalized based on user queries formulated in natural language. The expression of empathy/autonomy support is an additional characteristic that is experimentally manipulated and varies based on specific questions asked by a user. The intervention workflow is user defined (ie, a user may decide whether and how frequently to reuse the chatbot). However, our experimental setting included a single intervention (ie, the chatbot was used only once by each study participant).

In total, 211 Dutch-speaking adults living in Belgium (aged 18-60 years) and not vaccinated (yet) participated in an online experiment. The participants were panel respondents of the market research company Bilendi, Belgium. Their panel was recruited through various sources, such as the Bilendi website, banner media websites, member-get-member, third-party databases, and offline recruitment. To join the panel, respondents had to complete a sociodemographic questionnaire, approve a double opt-in where they confirmed they wanted to participate in Bilendi’s surveys, and accept their privacy policy. The participants were approached for the survey through personal email invitations and reminders. To thank the respondents for their contribution and encourage them to complete the online experiment, Bilendi granted points that the respondents could save up and swap for a prize later. The data collection was conducted in 2021, between May 28 and June 3, while daily new cases in Belgium were around 1400 [52], and around 54% of the adult population in Flanders and Brussels (where Dutch is an official language) were vaccinated against COVID-19 with at least 1 dose [53]. After the preintervention survey, the participants were redirected to the chatbot conversation (ie, the intervention) and then redirected to the postintervention survey. Due to logging problems, the responses of 15 (7.1%) participants could not be linked back to their IDs. Specifically, in the postintervention survey, all the participants were asked to manually fill out a code generated after their conversation with the chatbot. In 15 (7.1%) cases, a blank, false, or nonexisting code was filled out. Therefore, those responses could not be linked to the participants’ conversations. In similar future experiments, the logging should be automated to prevent such kind of data loss. Therefore, the final sample consisted of 196 (92.9%) participants (mean age 39.4, SD 9.4 years; n=112, 57.1%, females; see the sample characteristics in Table 1). At the time of surveying, 63.8% (n=125) of our participants had not yet received an invitation to be vaccinated against COVID-19, 33.7% (n=66) were invited and waited for the first shot, and 2.6% (n=5) were invited and refused. The vast majority of participants (n=186, 94.9%) declared they had not contracted COVID-19.

As experimental stimuli, we used VaccinChat, a chatbot providing COVID-19 vaccination information. VaccinChat was developed by the CLiPS (Computational Linguistics, Psycholinguistics and Sociolinguistics) research center associated with the Department of Linguistics of the Faculty of Arts of the University of Antwerp, Belgium. The chatbot is a RASA [54] artificial intelligence (AI)–equipped frequently asked questions (FAQ) chatbot, trained to classify user questions provided in open text fields into predefined answer intents, and it has been publicly available as a development version. It does not consider a specific conversation flow but answers questions in series. The final version is trained on 12,883 questions referring to 181 answers [55].

The experiment had a 2 (chatbot empathy/autonomy support expression: present vs absent) × 2 (endorser credibility: expert vs layperson) between-subject design. The participants were randomly assigned to the experimental conditions, however, striving for equal gender distribution. The final sample sizes were N_present=101 and N_absent=95 across the empathy/autonomy support expression condition and N_expert=111 and N_layperson=85 across the endorser credibility condition (experimental cell sizes: N_{empathy/autonomy present, expert}=57, N_{empathy/autonomy present, layperson}=44, N_{empathy/autonomy absent, expert}=54, N_{empathy/autonomy absent, layperson}=41). No significant differences in initial vaccination intention occurred across the experimental condition (empathy/autonomy support expression condition: t₁₉₄=0.336, P=.74; endorser credibility condition: t₁₉₄=0.719, P=.47). In addition, no significant differences occurred in demographics for empathy/autonomy support expression condition (gender: χ²₁=3.296, P=.07; age range: χ²₄=7.266, P=.12; education: χ²₄=8.367, P=.08). For endorser credibility condition, there were no significant differences in age and education (age range: χ²₄=1.628, P=.80; education: χ²₄=1.719, P=.79), but the percentage of women was higher in the expert condition (n=71 vs 41, 64.0% vs 48.0%, χ²₁=4.863, P=.03). To ensure the sample characteristics would not disturb the tested effects, we involved gender, age range, education, and initial vaccination intention as covariates in all analyses.

The participants first received an invitation mail with the survey link randomly redirecting them to the 4 experimental conditions. After the participant provided informed consent, the preintervention survey started with demographics and a question on COVID-19 vaccination status, initial COVID-19 vaccination intent, and personal experience with COVID-19. Next, the participants were informed about the chatbot, they received the related instructions and time recommendations, and they were redirected to the chatbot and asked to start the conversation. We encouraged the participants to submit their questions and concerns about COVID-19 vaccinations. The instruction was as follows:

Before starting a conversation with the chatbot, the participants read a brief endorsement claiming, “The chatbot gives a clear answer to the questions and concerns about the COVID-19 vaccines,” and that the chatbot is “particularly on point.” In the expert endorser condition, this endorsement was provided by “the well-known vaccinologist, Prof. Dr. Pierre Van Damme.” In the layperson endorsement condition, the endorsers were described as “a group of Flemings with questions about vaccines, including vaccine doubters.”

The chatbot worked differently depending on the empathy/autonomy support expression condition. Namely, in the present empathy/autonomy support expression condition, the chatbot displayed additional phrases to express its empathy and autonomy support. Specifically, the chatbot showed understanding of the user’s concerns (eg, “I understand your concern. This is ultimately about your own body.”), showed appreciation for the user’s knowledge (eg, “Good that you are aware of this.”), supported the user’s autonomy regarding the decision to get vaccinated (eg, “Whether or not you vaccinate your children, when possible, remains, of course, a personal choice.”), showed interest in the user’s situation (eg, “I hope with you that we can soon pick up life again in all its glory.”), and comforted the user by referring to the social context (eg, “There are many people who are currently facing this question.”). See all phrases appearing in the predefined set of the chatbot's answers in Table 2.

These phrases (Table 2) were removed in the absent empathy/autonomy support expression condition.

The empathy/autonomy support phrases were attached to predefined chatbot answers (called adapted answers). The occurrence of those answers in a conversation depended on the questions asked by each participant. In the actual conversations made with the participants in our study, the mean share of the adapted answers (ie, the ones belonging to the predefined set of the chatbot's answers differing between the high and low empathy/autonomy support expression condition) was 63% (SD 0.2349). To ensure that the share of adapted answers for empathy/autonomy support expression at the participant level would not disturb the tested effects, we involved it as an additional covariate in all analyses.

The most frequently asked questions were similar to those reported by Buhmann et al [55], such as out-of-domain questions, general COVID-19 questions that do not consider vaccination (eg, regulations), questions about side effects of the vaccines, and questions about who will get their vaccine when and where. User questions that deal with trust issues mainly focused on vaccines in general, science, and governmental institutes, and policies. The average number of user questions was similar across the empathy/autonomy support condition (mean_present 8.181, SD 7.364; mean_absent 8.787, SD 9.598; F_1,187=0.232, P=.63) and the endorser credibility condition (mean_expert 8.022, SD 9.688; mean_layperson 8.945, SD 6.674; F_1,187=0.548, P=.46). In addition, the average length of the user questions was similar across the conditions (empathy/autonomy support condition: mean_present 8.327, SD 4.403; mean_absent 7.669, SD 3.043; F_1,187=1.401, P=.24; endorser credibility condition: mean_expert 7.633, SD 3.021; mean_layperson 8.363, SD 4.637; F_1,187=1.752, P=.19).

After their conversation with the chatbot, the participants had to use the conversation code they received from the chatbot at the beginning of the conversation to start the postintervention survey. Based on that code, we were able to link their answers to their IDs. Next, we measured the participants’ perceived autonomy during the conversation, CPI, and the intention to get vaccinated against COVID-19. They were debriefed that the chatbot gave different responses to different participants, but the information provided by the chatbot did not contain any factual falsehoods. Finally, the participants were asked for feedback, thanked, and received monetary compensation.

The Ethics Committee for the Social Sciences and Humanities of the University of Antwerp, Belgium (code of ethical approval: SHW_21_57) approved this research. Prior to the study, all respondents signed the informed consent form. All data were anonymously collected; we only kept track of an assigned response ID. All human experimentation guidelines were respected. Our participants received monetary compensation for their contribution to our online experiment.

First, we ran 2 ANCOVAs to check the main effects of chatbot empathy/autonomy support expression on chatbot evaluation separately for PUA and CPI as dependent variables. The empathy/autonomy support expression condition (present vs absent) and the endorser credibility condition (expert vs layperson) were factors, and the share of adapted answers for empathy/autonomy support expression, gender, age, education, and initial vaccination intention were covariates. No significant manipulation effects occurred under the following conditions:

In addition, the 2 manipulations had no interaction effects (for PUA: F_1,187=0.735, P=.39; for CPI: F_1,187=0.327, P=.57), providing no support for H2.

Next, we checked the role of actual answers provided by the chatbot in the relationship between the empathy/autonomy support expression and chatbot evaluation. Specifically, we checked the moderation effects of conversational quality variables. We ran a series of moderation analyses (PROCESS macro [58], model 1) with the empathy/autonomy support expression condition (dummy-coded as 1=present, 0=absent) as an independent variable, PUA as a dependent variable, and a conversation quality variable as a moderator. The endorser credibility condition (dummy-coded as 1=expert, 0=layperson), the share of adapted answers for empathy/autonomy support expression, gender (dummy-coded as 1=women, 0=men), age, education, and initial vaccination intent served as covariates. We included the endorser credibility condition as a covariate to ensure the differences in its share between the empathy/autonomy support expression condition would not disturb the tested effects. No moderation effects occurred with the following moderators: CA (B=0.396, SE 0.434, t₁₈₆=0.911, P=.36), CD (B=–0.119, SE 1.154, t₁₈₆=0.103, P=.92), and WN (B=–0.362, SE 0.679, t₁₈₆=.533, P=.59). However, in the moderation model with CF as a moderator (Figure 1; R²=0.145, F_9,186=3.507, P<.001, variance inflation factors [VIFs]<1.2), the interaction effect of CF and empathy/autonomy support expression was negative (B=–3.358, SE 1.235, t₁₈₆=2.718, P=.007), in line with H1. The conditional effect of empathy/autonomy support expression at the CF level of +1SD was negative (Figure 2; B=–0.405, SE 0.158, t₁₈₆=2.564, P=.011), while being nonsignificant at the mean level (B=–0.103, SE 0.113, t₁₈₆=0.914, P=.36) and the –1SD level (B=0.031, SE 0.123, t₁₈₆=.252, P=.80). The Johnson-Neyman cutoff value of CF was 4.0% (Table 4). The conditional effects of empathy/autonomy support expression at CF levels below the cutoff were nonsignificant (P>.10), while for levels above the cutoff, the effects were significantly negative (P<.05). These results indicate that when CF is higher than 4.0%, empathy/autonomy support expression decrease PUA, while empathy/autonomy support expression does not have an impact on PUA when CF is lower than 4.0%.

We used CF as a moderator in further analysis. Aiming to check the consequences of the CF × empathy/autonomy support expression interaction on behavioral outcomes, we ran 2 first-stage moderated mediation analyses (PROCESS macro [58], model 7, 5000 bootstrap samples). Both analyses used the empathy/autonomy support expression condition (dummy-coded as 1=present, 0=absent) as an independent variable, PUA as a mediator, and CF as a first-stage moderator. The set of covariates was the same as in the former analysis (VIF<1.2).

In the first analysis, CPI served as a dependent variable (Figure 3; R²=0.601, F_9,187=35.237, P<.001). The effect of PUA on CPI was positive (B=1.095, SE 0.074, t₁₈₇=14.719, P<.001), and the moderated mediation index was negative (index=–3.676, BootSE 1.614, 95% CI –6.697 to –0.102). The conditional indirect effect of empathy/autonomy support expression on CPI at the CF level of +1SD was negative (B=–0.443, BootSE 0.202, 95% CI –0.809 to –0.005), while 95% CIs included 0 at the mean level (B=–0.113, BootSE 0.124, 95% CI –0.346 to 0.137) and the –1SD level (B=0.034, BootSE 0.132, 95% CI –0.224 to 0.305).

In the second analysis, VIS served as a dependent variable (Figure 4; R²=0.086, F_9,187=2.194, P=.03). The effect of PUA on VIS was positive (B=0.138, SE 0.051, t₁₈₇=2.704, P=.008), and the moderated mediation index was marginally negative (index=–0.464, BootSE 0.302, 90% CI –0.978 to –0.021). The conditional indirect effect of empathy/autonomy support expression on VIS at the CF level of +1SD was marginally negative (B=–0.056, BootSE 0.036, 90% CI –0.118 to –0.002), while being nonsignificant at the mean level (B=–0.014, BootSE 0.018, 90% CI –0.044 to 0.014) and the –1SD level (B=0.004, BootSE 0.019, 90% CI –0.023 to 0.039).

The aim of our study was to improve the understanding of the role of the conversation-related context in the effects of expressing empathy/autonomy support using COVID-19 vaccination chatbots. Specifically, we checked the moderating role of 2 conversation context factors (ie, conversation quality and chatbot expertise cues). The results of our experiment suggest that chatbot empathy/autonomy support expression may harm user evaluation and behavioral intentions related to a chatbot and diminish its usefulness in promoting COVID-19 vaccinations when the conversation quality is low (as indicated by the higher levels of CF). However, we did not find the effects of chatbot expertise cues manipulated through chatbot endorser credibility.

We demonstrated that the effect of expressing empathy/autonomy support using the COVID-19 vaccination chatbot on PUA was negative for a high level of CF, while being nonsignificant for its mean and low levels. Moreover, we showed that this moderation effect has consequences on CPI and chatbot persuasiveness. The indirect effects of empathy/autonomy support expression on CPI and VIS through PUA were negative for a high level of CF, while being nonsignificant for its mean and low levels. These results add to the growing literature on the user response to chatbots (eg, [6,9,11,33,45,46,49]). Our results are in line with numerous previous studies indicating mixed or negative effects of chatbots' empathy [6,11], robots' capability to infer human mental states [10], robots' and conversational agents' anthropomorphic features [12,45], and chatbots' displayed emotions [46] on users' response to chatbots. Importantly, unlike these previous studies, our study was based on the measurement of the actual quality of conversation with a chatbot user instead of presenting cues or scenarios to study participants.

CF (ie, the chatbot replying ”I do not understand“) may represent the deficiency of actual chatbot empathy and, in turn, its limited support for user autonomy as the chatbot fails to resolve the request of the user. As such, higher levels of CF may indicate to the chatbot users that the chatbot's actual answers lack empathy and support for user autonomy. Considering this, the negative interaction of chatbot empathy/autonomy support expression and CF that we revealed is in line with the notion that chatbot anthropomorphism (here, in the form of empathy/autonomy support expression) may increase user expectations and sensitiveness for the quality of interaction [12,49]. Namely, a higher CF indicates an issue with empathy and support for user autonomy in actual chatbot answers, and users are more sensitive to this issue when chatbot empathy/autonomy support expression is higher. Although Pelau et al [59] demonstrated the positive relationship between perceived AI empathy and perceived AI interaction quality, we discerned the negative interaction effects of manipulated chatbot empathy/autonomy support expression and the actual conversation quality. Put differently, when CF is higher, the higher empathy/autonomy support expression increases the incongruity in empathy/autonomy support between the chatbot's actual answers and its declarations (expressions). This incongruity may produce negative user reactions [50,51]. This way, we shed new light on the existing chatbot research focusing on incongruities in chatbot perception [60]. Particularly, our results highlight that the positive effects of chatbot empathy, suggested by the CASA approach [43,44], diminish not only when a chatbot is too humanlike, as proposed by UVM theory [11], but also in the presence of incongruity between the declared empathy and the chatbot’s actual outputs.

In terms of Sundar's [61] framework of human-AI interactions, the incongruity in empathy/autonomy support between the actual answers and the chatbot’s declarations may represent the discrepancy between the cue route (based on visible attributes of the AI system, such as empathy expression) and the action route (based on the effects of human-AI collaboration, such as the actual quality of conversation). This way, our results suggest that those 2 routes may influence the user response to an AI-based system not only separately but also via their interplay. We applied these chatbot communication considerations to vaccination intention, showing that this incongruity may harm the persuasiveness of vaccination chatbots. This way, we have contributed to the literature on vaccination promotion through chatbots (eg, [1,2,39]).

No interaction effects of expressing empathy/autonomy support and the chatbot’s endorser credibility might be caused by our experimental setting, which involved extensive real interactions with the chatbot. In this case, the user response might be more influenced by the actual quality conversation, and therefore, expertise cues were discounted. Interestingly, endorser credibility also had no significant main effect on the user response.

Certain limitations of our study call for further research. First, our study used the Dutch-speaking population of Belgian adults who are demonstrated to have relatively positive attitudes toward vaccinations (eg, 84.0% agreement with ”Vaccines are effective“ [62]). In our sample, the majority of the respondents (77.8%) declared to be ”very likely“ to get vaccinated against COVID-19 as the initial intention. This might produce a ceiling effect and thus limit the observed effects of our chatbot on VIS. Therefore, future studies should cover populations with more negative vaccine attitudes (eg, Latvian adults, with 70.9% agreement with ”Vaccines are effective“ [62]).

Second, we were limited to a specific stage of the COVID-19 pandemic (ie, about 1 year after introducing the vaccines) and a specific preventable illness. Further research can extend the investigation of chatbot empathy/autonomy support effectiveness in later pandemic stages (eg, when COVID-19 became endemic) and other preventable illnesses (eg, influenza).

Third, we conceptualized and operationalized chatbot empathy and autonomy support together. Therefore, future studies examining the effects of chatbot empathy/autonomy -support expression should distinguish between those 2 constructs in experimental manipulation and measurements. Next, further studies may investigate deeper the mechanisms underlying the negative effects of the interaction of empathy/autonomy support expression and conversation quality, which were demonstrated in this paper. Specifically, whether those effects result from chatbot empathy being more likely to evoke an eerieness perception (as considered in UVM theory [11]) when the quality of the information provided by the chatbot is lower may be checked.

Fourth, we measured conversation quality instead of manipulating it. Although this approach enhances realism, as the quality of answers provided by chatbots may heavily depend on the interactions with their users, manipulating this variable should provide better causal evidence of its moderating role. Therefore, future experiments can compare conditions differing in the percentage of questions a chatbot fails to answer.

Lastly, our null result regarding the moderating role of expertise cues in the effect of expressing empathy autonomy support may result from our experimental setting, in which the participants were involved in extensive interaction with the chatbot. Perhaps, in the case of less engaging interactions, expertise cues play a larger role. This possibility should be examined in further research.

Our study underlined the pitfalls of making vaccination chatbots empathetic in specific conversation-related contexts, namely too low conversation quality. The results may guide vaccine policy makers and chatbot developers to be careful with communicating chatbot empathy/autonomy support unless its capability to address user questions is firmly proven. Therefore, introducing a chatbot's empathy/autonomy support expression should be preceded by assessing the quality of answers provided by the chatbot. For example, in their early stage of development, chatbots with a high level of CF should hesitate to express their empathy and support for user autonomy. However, it is difficult to keep up with information flow manually when information changes quickly, like in a pandemic. Consequently, there may always be user questions for which the chatbot does not yet know appropriate answers, ending up with fallback. Automatic text-mining tools or a model, such as ChatGPT, could be used to scrape new information from the web more easily. In addition, an instruction model, such as ChatGPT, could provide a multitude of exemplary user questions. These could be used as extra training data, speeding up and improving the chatbot’s continuous growth, thus avoiding or reducing the number of fallback answers and, eventually, allowing the effective use of chatbot empathy/autonomy support expression.