This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2410) Web-based and Mobile Health Interventions (3297) mHealth for Wellness, Behavior Change and Prevention (2871)

Published on in Vol 6, No 4 (2022): April

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/33974, first published .
Use and Effect of Embodied Conversational Agents for Improving Eating Behavior and Decreasing Loneliness Among Community-Dwelling Older Adults: Randomized Controlled Trial

Use and Effect of Embodied Conversational Agents for Improving Eating Behavior and Decreasing Loneliness Among Community-Dwelling Older Adults: Randomized Controlled Trial

Use and Effect of Embodied Conversational Agents for Improving Eating Behavior and Decreasing Loneliness Among Community-Dwelling Older Adults: Randomized Controlled Trial

Authors of this article:

Lean L Kramer1 Author Orcid Image ;   Lex van Velsen2, 3 Author Orcid Image ;   Jenna L Clark4 Author Orcid Image ;   Bob C Mulder5 Author Orcid Image ;   Emely de Vet1 Author Orcid Image
Article Authors Cited by (7) Tweetations (4) Metrics

    Original Paper

    1Consumption and Healthy Lifestyles, Wageningen University & Research, Wageningen, Netherlands

    2eHealth cluster, Roessingh Research and Development, Enschede, Netherlands

    3Biomedical Signals and Systems Group, University of Twente, Enschede, Netherlands

    4Center for Advanced Hindsight, Duke University, Durham, NC, United States

    5Strategic Communication Group, Wageningen University & Research, Wageningen, Netherlands

    Corresponding Author:

    Lean L Kramer, PhD

    Consumption and Healthy Lifestyles

    Wageningen University & Research

    Hollandseweg 1

    Wageningen, 6706 KN

    Netherlands

    Phone: 31 623412092

    Email: lean.kramer@wur.nl


    Background: Embodied conversational agents (ECAs) have been proposed as a promising interaction modality for the delivery of programs focused on promoting lifestyle changes. However, it is not understood what factors influence the health effects of ECAs or their use.

    Objective: We aimed to (1) identify whether ECAs could persuade community-dwelling older adults to change their dietary behavior and whether ECA use could decrease loneliness, (2) test the pathways to these effects, and (3) understand factors influencing the use of ECAs.

    Methods: A randomized controlled trial was conducted. The intervention group received access to the PACO service for 8 weeks. The waitlist group started PACO use after waiting for 4 weeks. Two primary outcomes (eating behavior and loneliness) were assessed via online questionnaires at intake, upon joining the waitlist, after 4 weeks, and after 8 weeks. The third primary outcome (use) was assessed via data logs. Secondary outcomes were measured at the same time points, via questionnaires or an optional interview.

    Results: In total, 32 participants completed the intervention. We found a significant correlation between use in minutes on the one hand, and perceived usefulness (r=0.39, P=.03) and enjoyment on the other (r=0.38, P=.03). However, these did not predict use in the full regression model (F2,29=1.98, P=.16, R2=0.12). Additionally, PACO use did not lead to improvement in eating behavior (χ22=0.34, P=.85) or a decrease in loneliness (χ22=0.02, P=.99).

    Conclusions: Our study did not provide any concluding evidence about factors that are linked to the use or health effects of ECAs. Future service design could benefit from either creating a functional design catering to the predominant stage in the precaution adoption process model of the targeted population, or by personalizing the service based on an intake in which the end user’s stage is determined.

    Trial Registration: ClinicalTrials.gov NCT04510883; https://clinicaltrials.gov/ct2/show/NCT04510883

    International Registered Report Identifier (IRRID): RR2-10.2196/22186

    JMIR Form Res 2022;6(4):e33974

    doi:10.2196/33974

    Keywords

    eHealth; online intervention; embodied conversational agent; lifestyle change; older adult; user experience; eating habits; eating behavior 


    Background

    Embodied conversational agents (ECAs) have been proposed as a promising interaction modality for the delivery of programs focused on promoting lifestyle changes [1], such as physical activity [2-4] and nutrition [5,6], and preconception care [7,8]. ECAs often have a human-like appearance and communicate via prewritten dialogue. They also have the ability to establish and maintain an empathic relationship by using empathic behavior, both verbal, via text or speech, and nonverbal, via facial and gaze expressions and hand and body gestures [9]. These behaviors may make them more engaging than traditional eHealth interventions [10,11]. Results are promising, as ECA interventions have been found to be easier to use [5] and used more frequently [5,11-13] than interventions without an ECA. Nonetheless, ECA use does decline over time, limiting long-term health effects [1,14-17]. Moreover, it is unknown what factors influence use of an ECA. When designing an ECA, designers are advised to select the right role for the ECA, combine the most important personality characteristics, and use informational, nonjudgmental language [18]. In addition, a scoping review identified which use-related factors were assessed when evaluating the effect of ECAs on promoting healthy lifestyles. These factors included usability and user satisfaction, further specified as factors including liking and trusting the ECA and the desire to continue using the ECA. However, evidence for the effect of these factors on ECA use is limited. Furthermore, there is scarce and inconclusive evidence for the health effects of ECAs and the pathways to these effects [1].

    In order to assess the pathways to effects and understand ECA use when evaluating an ECA, conceptual models can be used (as shown in Figure 1; further details were reported in the research protocol for this study [16]). The conceptual model explaining ECA use is based on existing human-computer interaction literature, including the technology acceptance model (TAM) [19]. The key variables in TAM are perceived usefulness and perceived ease of use. Systematic reviews have shown that these 2 variables typically explain about 40 percent of an individual’s intention to use a technology in a variety of contexts [20-22]. However, there is mixed evidence regarding whether intention predicts actual use [23,24]. Since actual use, rather than intention to use, is deemed necessary to achieve any health benefits, use is at the center of the conceptual model. Increased use is expected to improve the intensity of the relationship with the ECA, because of the capacity of ECAs to establish and maintain an empathic relationship. Usability and perceived usefulness are hypothesized to act as antecedents for use, whereas increased usability is expected to result in increased perceived usefulness.

    The conceptual model explaining health effects occurring after the use of an ECA starts with behavioral change techniques (see Multimedia Appendix 1), which are expected to lead to an improvement in the 3 basic psychological needs: autonomy, competence, and relatedness. [25]. Ultimately, improved health behaviors will lead to a better quality of life. This model is primarily based on self-determination theory [25] and has an explorative character. By contrast, the classification system of Teixeira et al [26,27] is used to form hypotheses to explain which techniques improve which needs. Hence, the objectives of this study were to (1) identify whether ECAs could persuade community-dwelling older adults to change their dietary behavior and decrease their loneliness, (2) assess the pathways to these effects, and (3) understand ECA use.

    ‎
    Figure 1. Conceptual models explaining embodied conversational agent (ECA) use and health effects.
    View this figure

    Study Design

    This study used a randomized controlled trial design. Participants in the first cohort received access to the 8-week intervention immediately, while participants in the second cohort served as a control group, receiving access to the intervention after being placed on a 4-week waiting list.

    Participants and Procedure

    We aimed to include a total of 60 participants with a 1:1 ratio of participants per cohort. Participants were deemed eligible if they were aged 65 years or older, not in paid employment, and lived alone and independently at home. In addition, participants needed to speak Dutch, be able to use a tablet or computer by themselves, and have a wireless internet connection. The project members recruited participants via research panels, flyers, newspapers, and social media. After providing informed consent, the participants were invited to complete the intake questionnaire. They were asked to report their demographics (gender, age, educational level, health conditions, risk of malnutrition [28], and eHealth literacy [29]), their possession of a device to use for the study, and their motivation to participate. All participants were asked to complete the baseline questionnaire (T0) after creating an online account and complete another questionnaire after 4 (T1) and 8 (T2) weeks of use. Participants in cohort 2 were asked to complete an additional waitlist questionnaire (Tw) 4 weeks before T0. In the last questionnaire, participants were asked whether they were open to an interview by phone.

    Intervention

    The intervention, PACO, is a web-based eHealth service in which 2 ECAs engage in dialogue with an older adult to provide motivation for improving eating behavior and decreasing loneliness. The service consists of 5 modules, each one applying a different behavioral change technique (Figure 2, Multimedia Appendix 1). The user can engage in dialogue with Herman (the cook, who provides nutritional advice) and Ellen (the peer, who provides social advice). The ECAs are represented as 2D humans in cartoon style, are not animated, and use text as the means of communication. During the onboarding process, the ECAs introduce themselves and explain the PACO program.

    ‎
    Figure 2. PACO home screen.
    View this figure

    Outcomes

    The primary outcomes include use of the service, eating behavior, and loneliness. Use was assessed via log data collected on the PACO back end. Eating behavior was self-assessed by 3 open questions about the previous day’s fruit, vegetable, and liquid intake and loneliness was assessed via a validated questionnaire (Table 1 shows further details). The experience and the willingness to pay for PACO were measured via a self-compiled scale. All other outcomes were measured via validated online questionnaires. In an interview of approximately 30 minutes, participants were asked further questions about their experiences with PACO and any behavioral changes.

    Table 1. Study outcomes measured via questionnaires in each study phase.
    OutcomeScaleTwT0T1T2
    Use-related outcomes

    		Relationship with ECAaRapport scale [30-32]N/Ab✓✓✓

    		UsabilitySystem usability scale [33]N/AN/AN/A✓

    		EnjoymentAffect scale [34]N/AN/AN/A✓

    		AestheticsClassic aesthetics [35]N/AN/AN/A✓

    		Privacy concernsConcern for privacy scale [36]N/AN/AN/A✓

    		ControlActive control [37]N/AN/AN/A✓

    		Perceived usefulnessPerceived usefulness scale [19,38]N/AN/AN/A✓
    Health-related outcomes

    		Eating behaviorN/A✓✓✓✓

    		LonelinessDe Jong Gierveld loneliness scale [39]✓✓✓✓

    		Quality of lifeBrief older people’s quality of life questionnaire [40]✓✓✓✓

    		Autonomy, competence, and relatednessBasic psychological need
    satisfaction and frustration scales [41-43]    		✓✓✓✓
    Other outcomes

    		ExperienceN/AN/AN/A✓N/A

    		Willingness to payN/AN/AN/AN/A✓

    aECA: embodied conversational agent.

    bN/A: not applicable.

    Data Analyses

    We created a single score for each scale and checked the test assumptions. Due to the violation of the linearity assumption, we deviated from the original protocol by using nonparametric tests. Relationships between demographics and the main study outcomes were calculated using Spearman ρ and, for gender, Mann-Whitney U. Differences between Tw and T0, and in health-related outcomes over time, were compared using the Friedman test. Differences in the strength of the relationship with the ECA over time were compared with a repeated-measures ANOVA. Spearman ρ was used to calculate the correlations between use- and health-related outcomes. Linear regression analysis was used to calculate the multivariate relationships between use, eating behavior, loneliness, and significant outcomes. The statistical significance level was P<.05. Recordings of the interviews were transcribed and thematically analyzed by LLK and BCM.

    Ethics Approval

    This study was preregistered at ClinicalTrials.gov (NCT04510883) and approved by the medical ethics committee of Wageningen University (number NL73121.081.20). We refer to the study protocol article for all details on the protocol, the development process of the intervention, and the conceptual models [16].


    Drop-out, Baseline Characteristics, and Motivation

    In total, 51 participants met the inclusion criteria. Nineteen participants did not use the PACO service for 14 consecutive days and were treated as dropouts. Among participants who dropped out, 7 did not respond to emails or telephone calls, 3 dropped out due to illness, 3 due to lack of time, 2 due to lack of motivation, 2 due to difficulties with the service, and 1 due to internet issues. Eight participants who dropped out had created an account, of whom 4 had completed T0. The mean age of the 32 participants was 73.00 years (SD 5.33, range 65-85); 18 (56%) were women. In total, 12 (38%) had completed high school or an associate degree and 19 (59%) had completed college or university. The mean eHealth literacy score was 29.25 (SD 4.36, range 15-34), and the risk of malnutrition was 9.69 (SD 1.35, range 7-11). None of the demographic characteristics were significantly associated with use, eating behavior, or loneliness, and there were no significant differences in health-related outcomes between Tw and T0. During intake, participants stated that they were mainly motivated to participate because they were interested in research and in new developments and thought it was important to contribute. Some participated because they were interested in nutrition and wanted to stay healthy or improve their habits.

    Health Effects

    The ECAs were not able to persuade users to change their fruit, vegetable, or liquid intake (χ22=0.34, P=.85) or decrease loneliness (χ22=0.02, P=.99). There were also no significant differences over time in quality of life (χ22=2.99, P=.22), autonomy (χ22=0.34, P=.85), competence (χ22=2.32, P=.31), or relatedness (χ22=2.46, P=.29). Table 2 shows all descriptive health outcomes.

    During the interviews, most participants indicated that they thought they had a healthy diet. Nonetheless, a majority mentioned that the food diary helped them to become aware of their food intake. Some people were even shocked by the observation that they had such a fixed eating pattern and described PACO as a wake-up call. About half of the participants mentioned that they did introduce changes into their diet, such as cooking with more fresh ingredients, baking bread, eating more fruits and vegetables, and eating less meat. With respect to loneliness, most participants mentioned that they already had ample social contacts, even though some stated that they were feeling rather lonely. Apart from the unfortunate timing of the pandemic, 4 participants mentioned making changes in their social network because of PACO. For example, 1 participant created a list of everyone he knew and contacted them occasionally. Also, the chat connected a few people with each other and resulted in one-on-one contacts.

    Table 2. Descriptive health outcomes.

    		ScaleTw, mean (SD)T0, mean (SD)T1, mean (SD)T2, mean (SD)
    Eating behavior0-300237.04 (45.33)215.84 (72.12)215.70 (65.92)223.01 (71.28)
    Loneliness1-52.27 (1.71)2.47 (1.78)2.62 (1.91)2.44 (1.92)
    Quality of life13-6554.93 (4.93)56.09 (5.60)55.47 (6.56)54.78 (5.85)
    Autonomy1-54.11 (0.42)3.99 (0.54)4.05 (0.60)4.07 (0.56)
    Competence1-54.24 (0.37)4.05 (0.58)4.19 (0.52)4.22 (0.57)
    Relatedness1-54.21 (0.38)4.33 (0.56)4.31 (0.53)4.34 (0.51)

    Pathways to Effects

    Following our conceptual model for health, we expected to find a significant correlation between minutes spent on the different modules and eating behavior. However, this was not the case (P>.05, see Table 3 for all correlations). With respect to the other pathways, we found that competence correlated with eating behavior (r=–0.38, P=.03) and that it predicted eating behavior over time (F1,30=4.30, P=.047, R2=0.13). Quality of life (r=–0.60, P<.001), autonomy (r=–0.38, P=.03), relatedness (r=–0.59, P<.01), and number of chat messages (r=0.72, P=.03) correlated with loneliness, but did not predict loneliness (F4,8=1.32, P=.40, R2=0.14).

    Table 3. Spearman correlations for health-related outcomes and PACO modules.
    VariableEating behaviorLonelinessQuality of lifeAutonomyCompetenceRelatednessFood diaryGoalsRecipesStoriesChat
    Eating behavior

    		Spearman correlation10.10–0.21–0.11–0.38–0.010.190.12–0.270.040.01

    		P value—a.57.28.57.03.94.30.53.13.811.00
    Loneliness

    		Spearman correlation0.101–0.60–0.38–0.16–0.590.01–0.09–0.23–0.130.72

    		P value.57—<.001.03.39<.001.98.62.21.48.03
    Quality of life

    		Spearman correlation–0.21–0.6010.750.470.670.130.200.140.36–0.45

    		P value.26<.001—<.001.007<.001.48.28.45.046.22
    Autonomy

    		Spearman correlation–0.11–0.380.7510.550.600.09–0.02–0.030.16–0.31

    		P value.57.03<.001—.001<.001.62.92.88.37.41
    Competence

    		Spearman correlation–0.38–0.160.470.5510.430.06–0.32–0.24–0.010.06

    		P value.03.39.007.001—.014.74.08.18.99.89
    Relatedness

    		Spearman correlation–0.01–0.590.670.600.4310.120.130.200.24–0.29

    		P value.94<.001<.001<.001.01—.51.49.28.20.45
    Food diary

    		Spearman correlation0.190.010.130.090.060.1210.02–0.14–0.13–0.27

    		P value.30.98.48.62.74.51—.92.46.49.48
    Goals

    		Spearman correlation0.12–0.090.20–0.02–0.320.130.0210.340.44–0.37

    		P value.53.62.28.92.08.49.92—.06.01.33
    Recipes

    		Spearman correlation–0.27–0.230.14–0.03–0.240.20–0.140.3410.11–0.21

    		P value.13.21.45.88.18.28.46.06—.54.59
    Stories

    		Spearman correlation0.04–0.130.360.16–0.010.24–0.130.440.1110.03

    		P value.81.48.046.37.99.20.49.01.54—.93
    Chat

    		Spearman correlation0.010.72–0.45–0.310.06–0.29–0.27–0.37–0.210.031

    		P value1.00.03.22.41.89.45.48.33.59.93—

    aNot applicable.

    Understanding ECA Use

    Use of PACO and Trends Over Time

    On average, participants logged in 39.97 times (SD 37.38, range 10-197). Minutes per week decreased from a median of 69.66 in week 1 to 21.57 minutes in week 8 (Figure 3). The Friedman test confirmed this decline over time, showing a significant difference in use between weeks (χ27=31.46, P<.001). The median time for using PACO was 15 h, 15 min, and 05 s. The average total time spent on PACO was 6 h, 30 min (SD 05 h, 54 min, 01 s), and the time spent per session was 11 h, 10 min (SD 05 h, 44 min). The average number of modules used per session was 2.39 (SD 0.34). The most time was spent on the food diary (85.45%), followed by the recipes (6.36%), goals (4.58%), and stories (3.61%). In total, 11 participants signed up for the chat. They sent a mean of 27.78 messages (SD 15.55, range 13-67). During the final interaction, the module used most often was the food diary (41.67%), followed by the chat (25.00%), goals (16.67%), recipes (12.50%), and stories (4.17%).

    ‎
    Figure 3. Minutes per week.
    View this figure
    Use-Related Outcomes

    Usability, aesthetics, privacy concerns, and perceived control were rated above the midpoint of the scale (Table 4). The enjoyment and usefulness of PACO were rated below the midpoint of the scale, and perceived usefulness was rated relatively low. In total, 30 (94%) participants indicated that they were not willing to pay for PACO. With respect to the amount they would be willing to pay, results were contradictory, with 28 (88%) not willing to pay anything, and 4 (13%) willing to pay €5 (US $5.50). Following our conceptual model for ECA use, we found that aesthetics correlated significantly with usability (r=0.44, P=.01) and enjoyment correlated with perceived usefulness (r=0.48, P=.005). Although we found that perceived usefulness (r=0.39, P=.03) and enjoyment (r=0.38, P=.03) correlated with use in minutes (Table 5 shows all correlations), in the full regression model, these did not predict use (F2,29=1.98, P=.16, R2=0.12).

    During the interviews, participants stated that although they read the module content, they did not truly engage and often reported that the content was not helpful. For example, participants listened to stories and read recipes, but did not act. In some cases, this was due to issues of tone, such as storytellers being seen as patronizing or the discomfort of endorsing dining alone. In other cases, such as the chat, participants simply did not wish to speak to people they did not know, or, if they did do so, the conversations felt shallow.

    Table 4. Descriptive use outcomes.

    		ScaleOutcome, mean (SD)
    Usability0-10064.53 (17.98)
    Enjoyment1-73.26 (0.81)
    Aesthetics1-74.82 (1.21)
    Privacy concerns1-75.14 (1.28)
    Control1-74.78 (1.20)
    Perceived usefulness1-7 2.56 (0.99)
    Table 5. Spearman correlations for use and use-related outcomes.
    VariableUseRelationship with ECAaUsabilityPerceived usefulnessAestheticsEnjoymentPrivacy concernsControl
    Use

    		Spearman correlation1–0.13–0.050.390.340.380.30–0.01

    		P value—b.47.80.03.06.03.09.99
    Relationship with ECA

    		Spearman correlation–0.131–0.010.310.280.280.010.32

    		P value.47—.96.08.13.12.96.07
    Usability

    		Spearman correlation–0.05–.011–0.130.440.230.350.48

    		P value.80.96—.47.01.21.05.005
    Perceived usefulness

    		Spearman correlation0.390.31–0.1310.270.480.090.01

    		P value.03.08.47—.13.005.64.95
    Aesthetics

    		Spearman correlation0.340.280.440.2710.780.540.51

    		P value.06.13.01.13—<.001.001.003
    Enjoyment

    		Spearman correlation0.380.280.230.480.7810.320.38

    		P value.03.12.21.005<.001—.07.04
    Privacy concerns

    		Spearman correlation0.300.010.350.090.540.3210.48

    		P value.09.96.05.64.001.07—.005
    Control

    		Spearman correlation–0.010.320.480.010.510.380.481

    		P value.99.07.005.95.003.04.005—

    aECA: embodied conversational agent.

    bNot applicable.

    Relationship With the ECAs

    The strength of the relationship with the ECAs decreased over time (F1.72,53.33=4.22, P=.02; more details shown in Figure 4). Posthoc analysis with Bonferroni correction showed that the difference between T1 and T2 was significant (P=.047). Contrary to our expectation, the relationship did not correlate with use (r=–0.13, P=.47).

    During the interviews, most participants were neutral about the ECAs, or reported not having noticed them. Six participants mentioned that the ECAs made PACO easier to use, more engaging, or more enjoyable compared to plain text, or even described them as “fantastic.” On the other hand, 3 participants found the ECAs to be childish and unreal and the participants considered themselves too rational to regard the ECAs as actual people.

    ‎
    Figure 4. Relationship with the embodied conversational agents over time.
    View this figure

    This study used a randomized controlled trial to investigate the effectiveness, the pathways to effects, and the mechanisms that underlay the use of an ECA targeting eating behavior and loneliness among older adults. The results showed that neither the ease of use of the PACO service nor the user experience explained the extent to which it was used. Furthermore, the use of PACO did not result in improved fruit, vegetable, or liquid intake or reduced loneliness. Our findings might, on first sight, contradict our hypotheses, and add to the mixed evidence base on nutritional ECAs [5,6,44]. On the other hand, we can also take these results as valuable lessons for the future design of eHealth services.

    Participants did become more aware of their eating behavior due to the self-monitoring tool, and thus also became more aware of behaviors they could improve. In the terms used in the precaution adoption process model (PAPM) [45], they were “deciding about acting.” However, in the interviews, participants expressed high self-perceived health and no need for change. This suggests users might well have been in the stage of “decided not to act.” It is known that the PAPM stage plays a significant role in the perceived persuasiveness of different behavioral change techniques [46]. Hence, different needs should have been nurtured in our participants, as they were still in an earlier stage of the model. If this was the case, then the design of future services could benefit from either creating a functional design catering toward the predominant stage of the targeted population or personalizing the service based on an intake process that considers the stage of the end user.

    To our knowledge, we are among the first to study factors that help understand ECA use. Surprisingly, we found that the use of PACO could not be explained by its usability, privacy concerns, perceived usefulness, or level of enjoyment. Furthermore, positive ratings on aesthetics and perceived control were not associated with time spent using PACO, although these factors did have a positive correlation with usability. Instead of arguing that these factors are not relevant for the development of an ECA, we argue that a certain threshold might be necessary for a service to be used. This is in line with other work on ECAs among older adults, which has shown that technical problems have a negative impact on use, adaptiveness, usefulness, and trust [47]. It has yet to be determined what factors positively influence the use of an ECA. Instead of focusing on traditional use-related factors, as we did in our conceptual model explaining ECA use, future research might benefit from examining certain threshold scores in earlier stages of development. Furthermore, the PAPM might help understand ECA use. If users do not intend to change their behavior, for example, it can be expected that they will not engage with the ECA. Research has indeed shown that this is true for the adoption of nutrition and fitness apps among the general population [48].

    This study has limitations. First, we received a very low response to our flyers, social media posts, and advertorials. Newspaper interviews and phone calls to potential participants by the research panels resulted in a greater, yet still limited, response. Because of this nonresponse, we do not know why more older adults did not want to participate in this study. In total, 5 potential participants indicated that they did not want to participate on the consent form. One unintended effect of this method of obtaining informed consent could have been that people who were unable to provide consent were excluded. As a result of the small sample size, the overall power of this study was low. We consider that not measuring the PAPM stage was a second limitation of this study. We suspect that PAPM stage is a factor that might provide more insight into both the use and effectiveness of the service (ie, participants who use the service more frequently and report health-related effects might be more likely to act). Finally, the ongoing COVID-19 pandemic might have influenced our results. We did rewrite the content of PACO to match the current situation and focused on online alternatives for engaging in social interactions. Nonetheless, participants felt they were not able to be more socially active due to government restrictions. Indeed, loneliness increased in our target group during the pandemic [49]. This might have counteracted the decrease in feelings of loneliness that we expected.

    In conclusion, this study illustrates how to use a conceptual model to guide the evaluation of an ECA service in terms of both its level of use and its health effects, although it did not provide us with any conclusive evidence of its actual effectiveness. Nonetheless, our results provide valuable directions for future studies in this emerging field.

    Acknowledgments

    This work was supported by the Netherlands Association for Health Research and Development and its associated Create Health program (grant 40-44300-98-110). We would like to thank Dennis Hofs for developing the software, and MSc students Willine van den Brink and Iris Bijvelds for their assistance in performing the trial.

    Conflicts of Interest

    LvV works at Roessingh Research and Development, the company that developed PACO. There are no other conflicts of interest.

    ‎
    Multimedia Appendix 1

    The modules of the PACO service.

    DOCX File , 20 KB
    ‎
    Multimedia Appendix 2

    CONSORT e-HEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 372 KB
    1. Kramer LL, Ter Stal S, Mulder BC, de Vet E, van Velsen L. Developing Embodied Conversational Agents for Coaching People in a Healthy Lifestyle: Scoping Review. J Med Internet Res 2020 Feb 06;22(2):e14058 [FREE Full text] [CrossRef] [Medline]
    2. Bickmore TW, Caruso L, Clough-Gorr K, Heeren T. ‘It's just like you talk to a friend’ relational agents for older adults. Interacting with Computers 2005 Dec;17(6):711-735. [CrossRef]
    3. Bickmore TW, Pfeifer LM, Byron D, Forsythe S, Henault LE, Jack BW, et al. Usability of Conversational Agents by Patients with Inadequate Health Literacy: Evidence from Two Clinical Trials. Journal of Health Communication 2010 Aug 31;15(sup2):197-210. [CrossRef]
    4. Klaassen R, Bul KCM, Op den Akker R, van der Burg GJ, Kato PM, Di Bitonto P. Design and Evaluation of a Pervasive Coaching and Gamification Platform for Young Diabetes Patients. Sensors (Basel) 2018 Jan 30;18(2):1-27 [FREE Full text] [CrossRef] [Medline]
    5. Blanson HOA, Van der Boog PJM, Lindenberg J, Van der Mast CAPG, Neerincx MA, Zwetsloot-Schonk BJHM. An online lifestyle diary with a persuasive computer assistant providing feedback on self-management. Technol Health Care 2009;17(3):253-267. [CrossRef] [Medline]
    6. Bickmore TW, Schulman D, Sidner C. Automated interventions for multiple health behaviors using conversational agents. Patient Educ Couns 2013 Aug;92(2):142-148 [FREE Full text] [CrossRef] [Medline]
    7. Gardiner P, Hempstead MB, Ring L, Bickmore T, Yinusa-Nyahkoon L, Tran H, et al. Reaching women through health information technology: the Gabby preconception care system. Am J Health Promot 2013;27(3 Suppl):eS11-eS20 [FREE Full text] [CrossRef] [Medline]
    8. Jack B, Bickmore T, Hempstead M, Yinusa-Nyahkoon L, Sadikova E, Mitchell S, et al. Reducing Preconception Risks Among African American Women with Conversational Agent Technology. J Am Board Fam Med 2015;28(4):441-451 [FREE Full text] [CrossRef] [Medline]
    9. ter Stal S. Look who's talking: appearance of embodied conversational agents in eHealth. Enschede: University of Twente; 2021:1-272.
    10. Vardoulakis LR, Ring L, Barry B, Sidner CL, Bickmore T. Designing relational agents as long term social companions for older adults. Lect Notes Comput Sci 2012;7502:302. [CrossRef]
    11. Bickmore TW, Picard RW. Establishing and maintaining long-term human-computer relationships. ACM Trans. Comput.-Hum. Interact 2005 Jun;12(2):293-327. [CrossRef]
    12. Bickmore T, Gruber A, Picard R. Establishing the computer–patient working alliance in automated health behavior change interventions. Patient Education and Counseling 2005 Oct;59(1):21-30. [CrossRef]
    13. Hudlicka E. Virtual training and coaching of health behavior: example from mindfulness meditation training. Patient Educ Couns 2013 Jul 01;92(2):160-166 [FREE Full text] [CrossRef] [Medline]
    14. Provoost S, Lau HM, Ruwaard J, Riper H. Embodied Conversational Agents in Clinical Psychology: A Scoping Review. J Med Internet Res 2017 May 09;19(5):e151 [FREE Full text] [CrossRef] [Medline]
    15. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Translational Behavioral Medicine 2017 Jul 02;7(2):254-267 [FREE Full text] [CrossRef]
    16. Kramer LL, Mulder BC, van Velsen L, de Vet E. Use and Effect of Web-Based Embodied Conversational Agents for Improving Eating Behavior and Decreasing Loneliness Among Community-Dwelling Older Adults: Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2021 Jan 06;10(1):e22186 [FREE Full text] [CrossRef] [Medline]
    17. Cole-Lewis H, Ezeanochie N, Turgiss J. Understanding Health Behavior Technology Engagement: Pathway to Measuring Digital Behavior Change Interventions. JMIR Form Res 2019 Oct 10;3(4):e14052 [FREE Full text] [CrossRef] [Medline]
    18. Kramer LL, Blok M, van Velsen L, Mulder BC, de Vet E. Supporting eating behaviour of community-dwelling older adults: co-design of an embodied conversational agent. Design Health (Abingdon) 2021 Mar 04;5(1):120-139 [FREE Full text] [CrossRef] [Medline]
    19. Davis FD. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 1989 Sep;13(3):319. [CrossRef]
    20. Legris P, Ingham J, Collerette P. Why do people use information technology? A critical review of the technology acceptance model. Information & Management 2003 Jan;40(3):191-204. [CrossRef]
    21. King WR, He J. A meta-analysis of the technology acceptance model. Information & Management 2006 Sep;43(6):740-755. [CrossRef]
    22. Holden RJ, Karsh B. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010 Feb;43(1):159-172 [FREE Full text] [CrossRef] [Medline]
    23. Turner M, Kitchenham B, Brereton P, Charters S, Budgen D. Does the technology acceptance model predict actual use? A systematic literature review. Information and Software Technology 2010 May;52(5):463-479. [CrossRef]
    24. Wu J, Du H. Toward a better understanding of behavioral intention and system usage constructs. European Journal of Information Systems 2017 Dec 19;21(6):680-698. [CrossRef]
    25. Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 2000 Jan;55(1):68-78. [Medline]
    26. Teixeira P, Marques M, Silva M, Brunet J, Duda J, Haerens L, et al. A classification of motivation and behavior change techniques used in self-determination theory-based interventions in health contexts. Motivation Science 2020 Dec;6(4):438-455. [CrossRef]
    27. Teixeira PJ, Silva MN, Mata J, Palmeira AL, Markland D. Motivation, self-determination, and long-term weight control. Int J Behav Nutr Phys Act 2012;9:22 [FREE Full text] [CrossRef] [Medline]
    28. Nestlé. Self-MNA Mini Nutritional Assessment. Société des Produits Nestlé SA 2012:1-2 [FREE Full text] [CrossRef]
    29. van der Vaart R, van Deursen AJ, Drossaert CH, Taal E, van Dijk JA, van de Laar MA. Does the eHealth Literacy Scale (eHEALS) Measure What it Intends to Measure? Validation of a Dutch Version of the eHEALS in Two Adult Populations. J Med Internet Res 2011 Nov 09;13(4):e86. [CrossRef]
    30. Gratch J, Wang N, Okhmatovskaia A, Lamothe F, Morales M, van der Werf RJ, et al. Can virtual humans be more engaging than real ones? In: Jacko JA, editor. Human-Computer Interaction. HCI Intelligent Multimodal Interaction Environments. Berlin, Heidelberg: Springer; 2007:286-297.
    31. Gratch J, Wang N, Gerten J, Fast E, Duffy R. Creating Rapport with Virtual Agents. In: IVA 2007. Berlin, Heidelberg: Springer; 2007:125-138.
    32. Acosta JC, Ward NG. Achieving rapport with turn-by-turn, user-responsive emotional coloring. Speech Communication 2011 Nov;53(9-10):1137-1148. [CrossRef]
    33. Brooke J. SUS: A quick and dirty usability scale? In: Usability evaluation in industry. Reading, UK: Redhatch Consulting Ltd; 1996:189-194.
    34. Cheung W, Chang MK, Lai VS. Prediction of Internet and World Wide Web usage at work: a test of an extended Triandis model. Decision Support Systems 2000 Dec;30(1):83-100. [CrossRef]
    35. Lavie T, Tractinsky N. Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human-Computer Studies 2004 Mar;60(3):269-298. [CrossRef]
    36. Chellappa RK, Sin RG. Personalization versus Privacy: An Empirical Examination of the Online Consumer’s Dilemma. Inf Technol Manage 2005 Apr;6(2-3):181-202. [CrossRef]
    37. Liu Y. Developing a scale to measure the interactivity of websites. JAR 2003 Jun 01;43(2):207-216. [CrossRef]
    38. Heerink M, Kröse B, Evers V, Wielinga B. Assessing Acceptance of Assistive Social Agent Technology by Older Adults: the Almere Model. Int J of Soc Robotics 2010 Sep 4;2(4):361-375. [CrossRef]
    39. de Jong Gierveld J, van Tilburg T. A 6-Item Scale for Overall, Emotional, and Social Loneliness: Confirmatory Tests on Survey Data. Research on Aging 2006 Sep;28(5):582-598. [CrossRef] [Medline]
    40. Bowling A, Hankins M, Windle G, Bilotta C, Grant R. A short measure of quality of life in older age: the performance of the brief Older People's Quality of Life questionnaire (OPQOL-brief). Arch Gerontol Geriatr 2013;56(1):181-187. [CrossRef] [Medline]
    41. Chen B, Vansteenkiste M, Beyers W, Boone L, Deci EL, Van der Kaap-Deeder J, et al. Basic psychological need satisfaction, need frustration, and need strength across four cultures. Motiv Emot 2014 Nov 12;39(2):216-236. [CrossRef]
    42. Campbell R, Vansteenkiste M, Delesie LM, Mariman AN, Soenens B, Tobback E, et al. Examining the role of psychological need satisfaction in sleep: A Self-Determination Theory perspective. Personality and Individual Differences 2015 Apr;77:199-204. [CrossRef]
    43. van der Kaap-Deeder J, Vansteenkiste M, Soenens B, Loeys T, Mabbe E, Gargurevich R. Autonomy-Supportive Parenting and Autonomy-Supportive Sibling Interactions. Pers Soc Psychol Bull 2015 Sep 16;41(11):1590-1604. [CrossRef]
    44. Gardiner PM, McCue KD, Negash LM, Cheng T, White LF, Yinusa-Nyahkoon L, et al. Engaging women with an embodied conversational agent to deliver mindfulness and lifestyle recommendations: A feasibility randomized control trial. Patient Educ Couns 2017 Sep;100(9):1720-1729. [CrossRef] [Medline]
    45. Weinstein ND. The precaution adoption process. Health Psychol 1988;7(4):355-386. [CrossRef] [Medline]
    46. Oyebode O, Ndulue C, Mulchandani D, Zamil Adib AA, Alhasani M, Orji R. Tailoring Persuasive and Behaviour Change Systems Based on Stages of Change and Motivation. 2021 May Presented at: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems; 2021; Yokohama, Japan p. 1-19. [CrossRef]
    47. Stara V, Vera B, Bolliger D, Rossi L, Felici E, Di Rosa M, et al. Usability and Acceptance of the Embodied Conversational Agent Anne by People With Dementia and Their Caregivers: Exploratory Study in Home Environment Settings. JMIR Mhealth Uhealth 2021 Jun 25;9(6):e25891 [FREE Full text] [CrossRef] [Medline]
    48. König LM, Sproesser G, Schupp HT, Renner B. Describing the Process of Adopting Nutrition and Fitness Apps: Behavior Stage Model Approach. JMIR Mhealth Uhealth 2018 Mar 13;6(3):e55 [FREE Full text] [CrossRef] [Medline]
    49. van Tilburg TG, Steinmetz S, Stolte E, van der Roest H, de Vries DH. Loneliness and mental health during the COVID-19 pandemic: A study among Dutch older adults. The Journals of gerontology. Series B, Psychological Sciences and Social Sciences 2021 Aug 01;76(7):e249-e255 [FREE Full text] [CrossRef] [Medline]

    ‎
    ECA: embodied conversational agent
    PAPM: precaution adoption process model
    TAM: technology acceptance model

    Edited by A Mavragani; submitted 04.10.21; peer-reviewed by V Stara, S Orrange; comments to author 15.11.21; revised version received 05.02.22; accepted 19.02.22; published 11.04.22

    Copyright

    ©Lean L Kramer, Lex van Velsen, Jenna L Clark, Bob C Mulder, Emely de Vet. Originally published in JMIR Formative Research (https://formative.jmir.org), 11.04.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.