Digital health technologies, including chatbots and conversational artificial intelligence (AI) agents, have the potential to reshape health care delivery and patient outcomes [1,2]. Chatbots use natural language processing and AI to mimic human interactions and foster efficient communication between patients and health care providers. These digital tools can potentially transform medical care by aiding in patient screening, triage, and education, while simultaneously alleviating the strain on health care systems [3].

Although chatbots can potentially transform chronic disease management, evidence of their sustained adoption remains limited, particularly for autoimmune conditions [4,5]. Currently, most chatbot applications focus on oncology and mental health, with minimal customization for inflammatory rheumatic diseases [6,7]. Early diagnosis and treatment of inflammatory rheumatic diseases are critical for preventing irreversible joint damage, disability, and other complications [8]. Unfortunately, significant barriers, including rheumatologist shortages, high costs, and low public awareness, restrict optimal rheumatic care globally. Chatbots can address these challenges by facilitating patient education, self-triage, and access to rheumatology expertise. Despite the immense potential of chatbots to improve rheumatic disease management, there is minimal research on their real-world implementation and efficacy.

To address this gap, we evaluated patients’ perceptions and acceptance of a chatbot purpose-built for rheumatology. We aimed to provide insights into the successful adoption of chatbots in rheumatology care by assessing user attitudes, satisfaction, demographics, and the differences between the first and follow-up (FU) encounters. In the following section, we review the literature on conversational AI agents in health care, focusing on autoimmune inflammatory rheumatic diseases (AIIRDs). We then present the methodology and results of a survey evaluating patients’ acceptance of an AIIRD chatbot in a rheumatology clinic. Finally, we discuss the implications of our findings, limitations, and the future research needed to realize the potential benefits of thoughtfully designed chatbots for improving outcomes in autoimmune diseases.

The RE-AIM framework serves as a robust model for evaluating the impact of health interventions encompassing 5 pivotal dimensions: reach, effectiveness, adoption, implementation, and maintenance [32,33]. In our study, we used this framework to design a survey to meticulously assess the effectiveness, acceptability, and implementation processes of chatbots in health care settings. Specifically, the survey, detailed in Table 1, was constructed to scrutinize the various facets of participants’ interactions with the chatbot, including its effectiveness, user-friendliness, and potential applicability. The survey questions were structured to resonate with the dimensions of the RE-AIM framework, establishing a coherent link between the survey instrument and the framework, as elucidated below.

Q1 evaluated the chatbot’s prowess in responding satisfactorily to user inquiries, thereby gauging its effectiveness in addressing diverse concerns. Q3 assessed the comprehensibility of the information rendered by the chatbot to ensure effective interpretation and application in health care decisions. Q5 probed participants’ comfort in using chatbots for information retrieval after consultation, shedding light on their supplementary informational value. The categorization of rheumatological diagnoses into 8 groups measured the chatbot’s impact on disease screening and triaging, aligned with the effectiveness dimension.

Q2 measured the ease of navigation within the chatbot, which is a critical factor that influences user satisfaction and chatbot adoption feasibility. Q6 gauged participants’ willingness to engage with the chatbot in the future, especially if it aids in symptom identification and offers effective management guidance.

Q4 investigated participants’ comfort in using chatbots for pertinent information retrieval before the consultation, emphasizing chatbots’ potential in the preconsultation phase.

Q7, posed before the consultation (BC) and after the consultation (AC), explored participants’ comfort levels regarding chatbots making diagnoses based on their symptoms, unveiling any attitudinal shifts toward the chatbots’ diagnostic role after the consultation.

A pilot study was conducted from June to October 2022, with a predetermined sample size of 200 participants, which included 100 new cases and 100 FU cases. The sample size was determined based on previous studies [34,35]. Upon registration in the rheumatology outpatient clinic, a random sample of patients aged ≥17 years with a diagnosis of AIIRD or a referral for suspected AIIRD was selected. The participants were invited to participate in the survey, and verbal consent was obtained before administering the survey questions. Patients diagnosed by rheumatologists were classified into 3 groups: arthropathies, connective tissue diseases or vasculitides, and other diagnoses. Patient visits were classified into 2 types: first visit (FV), indicating the initial clinic visit, and FU visit, indicating return visits after the initial visit. The patient education level was classified as below university education, university degree, or higher.

SingHealth RheumConnect is a rule-based chatbot designed for internet-enabled devices. It was introduced in 2022 and serves as a valuable web-based resource for patients seeking treatment for rheumatology. The chatbot aims to alleviate resource and service deficiencies in rheumatology care by facilitating referrals and aiding patients in symptom management. Figure 1 and Figures S1-S7 in Multimedia Appendix 1 show the representative screenshots of the interface. SingHealth RheumConnect offers comprehensive information regarding various rheumatic diseases and arthritic conditions. The conditions covered were gout, systemic lupus erythematosus (lupus), myositis, osteoarthritis, psoriatic arthritis, rheumatoid arthritis, spondyloarthritis, systemic sclerosis, and vasculitides. This platform serves patients, caregivers, and the public. SingHealth RheumConnect also features an intuitive symptom checker to assess whether symptoms could indicate an autoimmune disease. It also connects users to essential services such as prescription renewals and appointment modifications. The chatbot is now being incorporated for normal use and is available 24/7 at no cost. Personal information is not collected during the advice-seeking interactions.

Statistical analyses were performed using Excel (Microsoft Excel 365, Microsoft Corporation) and SPSS software (version 28.0, IBM SPSS Statistics for Windows). Independent sample 2-tailed t tests or 1-way ANOVA were used to compare continuous variables, such as age and questionnaire scores. Chi-square tests were used to compare categorical variables such as sex, education, and diagnosis categories between the groups. A paired t test was conducted to compare the attitudes of the participants regarding the acceptability of chatbot diagnoses before and after clinical consultation. Specifically, this analysis compared the mean scores on the questionnaire items Q7 BC and Q7 AC. Q7 asked about comfort with the idea of a chatbot making a diagnosis based on the reported symptoms.

Linear regression analysis was performed to assess the factors associated with the Q6 score, which showed a significant difference between the FV and FU visit. Q6 asked about the intent to use the chatbot again if it helped determine the causes of the symptoms. The predictors entered into the regression model included sex (male or female), age (continuous), type of visit (FV vs FU), education (below university vs university degree or above), and diagnostic category (arthropathies vs connective tissue diseases or vasculitides vs other diagnoses). Q6 score was treated as a continuous dependent variable. Univariate linear regression was performed for the first time to examine the unadjusted associations between each predictor and Q6 score. Variables with a P value <.25 were selected for the final multivariate model entry. Multivariate linear regression using the enter method was conducted to evaluate the independent predictors of the Q6 score after adjusting for other variables in the model. Beta coefficients with 95% CIs were estimated. The variance inflation factor was examined to assess multicollinearity between predictors. Statistical significance was set at P<.05.

No written consent or ethical approval was obtained because the collected information did not include any personal data, thus not allowing retrospective identification of the survey participants.

In a cross-sectional study framed by the RE-AIM framework, we assessed the user satisfaction and acceptability of a health care chatbot among 200 patients, exploring potential variations across demographic strata, such as age, sex, and education. Our cohort, comprising 60.5% female participants, mirrored the demographics in Singapore [36]. Overall, our analysis revealed a favorable reception for the chatbot, as reflected by mean scores nearing 4 on a 5-point scale for most questionnaire items related to information retrieval and diagnosis via the chatbot.

We also aimed to elucidate how FV and FU patients, representing various stages of treatment, engaged in and viewed the chatbot. The results revealed key differences between the groups. Specifically, FU patients reported a significantly higher willingness to reuse the chatbot for symptom investigation than FV patients. This suggests that FU patients may become more receptive to chatbots for symptom analysis after gaining familiarity with their clinical context through ongoing care. Further, the survey indicated a favorable shift in patients’ attitudes toward the acceptability of chatbot-based diagnosis after interacting with physicians. Patients expressed greater openness to the chatbot and made a diagnosis based on their symptoms after the clinical encounter. This highlights that physician endorsements may help mitigate the initial concerns patients have regarding the chatbots’ diagnostic accuracy. These findings align with prior studies demonstrating that health chatbot acceptability can be affected by factors such as perceived accuracy of diagnosis, inability to conduct physical examinations, and patient preference for communication with physicians over chatbots [37,38]. Our observations imply that establishing patient confidence in the legitimacy of chatbots as diagnostic decision-making aids relies heavily on securing ongoing support and reassurance from health care providers as part of the care team [11].

We found that patients in all age groups had a similar acceptance of using a chatbot to search for information and as a diagnostic tool. Similarly, sex did not significantly influence the chatbot’s acceptability. These results suggest that chatbots could benefit patients at different stages of treatment as both a source of information and a diagnostic tool. Our findings align with those of a recent study by Iancu and Iancu [23], who found that age and sex did not appear to influence chatbot use, whereas subjective norms, health consciousness, and perceived convenience were influential factors. Furthermore, Chang et al [29] found that attitudes and subjective norms were positively related to individuals’ intentions to use medical chatbots. Taken together, it is evident that to maximize the adoption and use of chatbots, designers and developers should adopt user-centered approaches that address user concerns and problems [39].

Despite these insightful findings, this study had several limitations. The participant pool, mainly drawn from clinical settings, such as new referrals and FUs at rheumatology clinics, may not fully represent a broader population. The chatbot’s design, catering to Asian patients diagnosed with AIIRDs, may limit the generalizability of the results. Although structured, the RE-AIM model brought about challenges owing to its inherent intricacies. Moreover, the sampling process might have introduced a nonresponse bias, with only 200 (16.2%) of the 1232 patients providing consent to participate, which necessitates consideration. This study also overlooked the specific challenges faced by 31 participants aged >71 years. Although this group reported no explicit difficulties, the broader context of Singapore’s digital literacy programs aimed at seniors may have been a contributing factor.

For a more comprehensive understanding of patient acceptance of chatbots, future research could also delve into the socioeconomic and cultural factors that influence chatbot adoption, possibly by referencing broader digital literacy initiatives. Exploring strategies to enhance patient engagement, possibly through gamification or personalized user experiences, could also be considered in subsequent studies. Moreover, addressing challenges such as data privacy and the complexity of medical contexts is crucial for the integration of chatbots in health care. Multicenter studies with larger sample sizes are necessary to refine the accuracy and representativeness of our findings. Transitioning to machine learning–driven prediction models may address concerns regarding chatbot diagnostic accuracy [40]. Moreover, an education and training framework for health care providers encompassing informational sessions and hands-on training modules is essential for successful chatbot integration into health care practice. Finally, establishing standardized guidelines and fostering an iterative improvement process based on provider feedback can significantly contribute to the refinement and broader acceptance of chatbot interfaces in health care settings [41,42].

This study underscores the potential of chatbots, particularly those tailored for AIIRDs, to affect rheumatology care delivery positively. Our results revealed that patients who had FU visits showed a stronger propensity to re-engage with the chatbot, particularly when they were offered information on symptomatology. Sex, educational attainment, and specific diagnostic categories did not significantly influence chatbot acceptance. Furthermore, greater comfort with chatbot-mediated diagnoses was observed after physician consultations. This suggests that prior clinical experience and physician validation can significantly improve confidence in emerging digital tools. Although more rigorous investigations are warranted to fine-tune their integration into routine care, current evidence suggests that appropriately developed chatbots can fortify the nexus between patients and health care professionals, thereby improving the overall efficacy of rheumatology care.