Principal Findings
Nearly 60% (1669/2838) of all questions from users were on blood glucose, diet, GDM, and physical activity, and the chatbot was able to answer 88.51% (2512/2838) of all posed questions. The chatbot was most frequently used during the daytime, Monday through Friday, and most dialogues were short, containing 1-3 questions. However, the mean number of dialogues per week was 36 in the first period and 26.83 in the second period.
Most Frequently Asked Questions to Dina the Chatbot
Few prior studies have evaluated health chatbots based on log reviews [28]. Inkster et al [27] explored log data on self-reported symptoms of depression, using questionnaires integrated in their chatbot, Wysa, but with a different focus than ours. To the extent of our knowledge, our study is the first study aiming to explore what users ask a health chatbot integrated into a national health service platform by categorizing incoming questions. Dina the chatbot provides the opportunity to click on related themes and questions, “guiding” the user through the conversation. This might have increased conversation efficiency and influenced findings in this study by making some information more available for the users than other topics. Our results indicate that users mainly seek information with high relevance for their currently experienced issues related to GDM (questions regarding blood glucose levels and diet) as opposed to relevant information on future events like the postpartum period or contextual factors such as social health benefits. The information sought also seems to overlap with information available through the established antenatal health care program in Norway, as distribution of the most frequent questions mirrors the cornerstones in the treatment of GDM [14,15]. This indicates that the chatbot could serve as a low-threshold addition to the already-established health care service. The chatbot may enhance the treatment of GDM, promote stable blood glucose, and thereby prevent the development of adverse outcomes for the mother and the fetus. Qualitative studies have shown that women diagnosed with GDM may perceive a lack of personally adapted information, contributing to a feeling of insecurity [17,33] in which managing blood glucose measurements and changing diets are the main challenges [18]. Information provided from a chatbot can thus serve as a reminder or a confirmation for the user on already-received information from medical professionals [33] and not as a substitute for the traditional face-to-face consultation [39]. For users, adding technology like informational chatbots to the standard patient care may reduce insecurities [33,40] and potentially contribute to increased self-efficacy [19].
What Can User Behavior in the Chatbot Tell Us?
Despite some unanswered questions, we found that exploring user behavior in the chatbot will provide useful information for planning and organizing future antenatal care. A previous study on a comparable supportive chatbot developed for patients with breast cancer explored user behavior by asking a weekly question and by observing the retention rate among users [25]. As all data in our study were anonymous, we were unable to explore retention rate, and we treated each dialogue as the unit of analysis. We anticipated a higher frequency of use out of office hours, when medical professionals are less available. Surprisingly, the chatbot was most frequently used during the daytime, Monday through Friday. Even though a great advantage of chatbots is their 24-7 availability, frequent daytime use provides valuable insight for planning future antenatal care. To our knowledge, prior studies on health chatbots have not explored this issue before. The frequent use during the daytime may be a result of the users needing to quickly access information already provided for them by the health care service. With the timeframe of consultations often being limited [41], questions may arise before or after consultations [33]. The chatbot may provide reassurance for managing the condition of variable validity as an alternative to Google or other internet sources [33]. A chatbot may also provide answers to questions that appear too insignificant or embarrassing to ask health personnel directly [33], potentially reducing the barriers for contacting the health care service [33,42,43].
As there is currently a lack of a standardized methods for evaluating health chatbots, a comparison of chatbots performances may be challenging [25,28]. In general, metrics used to measure chatbot performance depend on which purpose the chatbot is designed to serve; still, most developers aim to keep the conversations short and effective [44]. Keeping this in mind, our results may prove an effective change in user interface in Dina version 2.0, as findings indicate that this version requires fewer conversational steps from the user, evidenced by both fewer questions from users on maneuvering in the chatbot and a decrease in dialogues containing 4-6 questions. This is supported by findings from a previous qualitative study, in which participants stated that they perceived Dina version 2.0 to be effective in providing answers [33]. However, short dialogues could also be an indication of users “giving up” and leaving the conservation; nonetheless, efficiency and the ability to provide a fast answer are important for the intention to use a chatbot [33].
Despite the fact that Dina the chatbot was integrated into the Norwegian digital health platform and made available for all pregnant Norwegian women between the 2 periods of registration, we found that the weekly mean number of dialogues was 36 in the first period and 26.83 in the second period. Although this change was not significant, it could be explained by a possible insufficient promotion of the chatbot among both pregnant women and health care personnel [33]. Previous studies have described several obstacles like organizational, economic, and knowledge barriers when implementing new technology in the health care service [45]. It is our view that the promotion of the chatbot should be a priority going forward to increase the chance of implementation of the chatbot in Norwegian antenatal care.
The Chatbot´s Ability To Answer and Need for Further Development
As the chatbot currently does not provide users with the opportunity to express if they are satisfied with the answer, we used the percentage of questions that the chatbot was unable to answer (fallback) as a measure of how well it operates. Our findings showed that a fallback message was given in 11.49% (326/2838) of all questions asked to Dina the chatbot. The goal is to keep the percentage of fallback as low as possible in order to meet user satisfaction [46]. In a previous study on conversational repair in a chatbot developed for customer service, the fallback percentage was reported to be 15% [47]. However, we did not consider “false positive” responses, where the chatbot seemingly provided an answer but not the answer the user sought. This would have provided more insight into the chatbots ability to answer and will be important to consider in future analyses. We categorized questions resulting in a fallback message to discover problem areas that need further development [48] and to satisfy our specific aim of determining what type of questions the chatbot is unable to answer. Our results (Figure 5) suggested an increase in the fallback category “GDM” between the 2 periods of registration. These are questions the chatbot should be able to answer, and calls for further training and development of content regarding information on GDM are warranted. Notably, we found that that number of questions that the chatbot did not understand decreased in the second period, which may be an indication of increased functionality in Dina version 2.0, as the chatbot may provide more options to click on related themes, thereby guiding the user in a more efficient way. Nearly 1 in 4 questions resulting in a fallback message were related to pregnancy in general, and this may be viewed as an expression of an interest from users and possibly serve as an idea for future development. Nonsensical questions represented 16.9% (55/326) of the overall questions resulting in a fallback message from the chatbot, and one could therefore consider the real fallback rate for the chatbot to be lower than 11.5% if one disregarded these questions.
There are several challenges when it comes to integrating this type of new technology into the established health care [39]. Studies outline several reasons as to why users of health technology tools such as chatbots can lose interest, among them being frustrations with the technology and losing the in-person contact with the caregiver [49,50]. A chatbot is not a finished product once it is deployed, and it is important to continuously monitor and add information to improve the chatbot’s ability to function optimally [51]. By highlighting areas in Dina the chatbot that need improvement, we hope this study may serve as a contribution for further improvement and implementation.
Strength and Limitations
The strengths of this study are its use of the chatbot log data from 2 different periods during the course of continuous development and maintenance of the chatbot [52]. Utilization of user data is a cost-effective approach, providing important insight needed for further development. This study also has some limitations. We were unable to assess if users were women with GDM; however, the chatbot could also be a resource for health personnel or partners or next of kin. As all questions to the chatbot are displayed as dialogues in the chatbot’s log, we were not able to stratify our analysis according to which method the user decide upon (ie, free text, theme buttons, or a mix of the 2). A registration of users’ preferred way of asking questions should be considered in the future development of the chatbot. Furthermore, we could not identify if the user visited the chatbot once or several times. There is currently no way for the users to express whether they received the answer to their question in the dialogue, and obtaining this information would have been a useful addition to the fallback percentage acquired in analyses. Moreover, it would be useful to explore “false positive” responses by the chatbot (the chatbot provides an answer but not the answer the user is seeking). This would have provided more insight into the functionality of the chatbot and will be considered in future studies. The chatbot is currently only available in Norwegian language, limiting the external validity of the knowledge. In addition, not all women may feel comfortable trusting a chatbot on health issues, which might have potentially excluded some women and introduced selection bias into our study.
Implications
Our findings indicate that users seek information on topics relevant to them at the time, such as blood glucose, diet, and physical activity, and that the most frequently asked questions mirror the cornerstones of GDM treatment. This may indicate that the chatbot is used to quickly access information already provided to users by the health care service, but the chatbot offers a low-threshold way to access that information.
Furthermore, results indicate that Dina version 2.0 guides the user in a more efficient way. However, the low mean number of dialogues per week in the second period (26.83) suggests further efforts should be put into promoting and integrating the chatbot into Norwegian antenatal care. We view our findings as potentially relevant to future development of informational and supportive health chatbots. The authors finds a low-threshold design is an advantage, as this will provide easy access to information the user is also provided through other channels in the health service to further support self-efficacy. As our society quickly becomes more digital, there is a call for the health care service to keep up with the rapid development [36]. We see the need for an informational tool like Dina the chatbot to contribute to increased self-efficacy and coping with GDM, considering the rising prevalence of the condition [2,4,5]. The next step for us with Dina the chatbot would be to continue the ongoing work of further development and to improve promotion to increase its use. We believe that when chatbots are a more integrated part of the health care service, they may serve as a positive contribution to antenatal care.
Conclusions
The majority of posed questions pertained to blood glucose, diet, the GDM diagnosis, and physical activity, and the chatbot was able to answer about 9 out of 10 of all questions from the users. The most frequent use was during the daytime, Monday through Friday, and the majority of dialogues were short, containing 1-3 questions. However, the mean number of dialogues per week was 36 in the first period and 26.83 in the second period.