An evaluation study of the implementation of the DOTTS in daily practice with a mixed methods design was performed. As methods, a questionnaire survey and a qualitative focus group discussion were used.

All 9 hospitals that implemented the DOTTS before September 1, 2019, were included in this study. Of the 9 hospitals, 2 were academic hospitals, 5 were teaching hospitals, and 2 were nonteaching hospitals in the Netherlands. Participating hospitals were (1) Erasmus MC Rotterdam, (2) Leiden University Medical Center Leiden, (3) Jeroen Bosch Hospital 's-Hertogenbosch, (4) Antonius Hospital Utrecht, (5) OLVG Amsterdam, (6) Amphia Hospital Breda, (7) Elisabeth Tweesteden Hospital Tilburg, (8) Tjongerschans Hospital Heerenveen, and (9) IJsselland Hospital Capelle aan de IJssel.

In all participating hospitals, the DOTTS was implemented and introduced into routine care. Implementation strategies of the DOTTS were designed for each hospital separately. For this aim, each hospital formed an implementation team with stakeholders. In each hospital, stakeholders involved were at least one nurse and one other care professional (ie, midwife, obstetrician, or obstetrician in training). In most hospitals, implementation teams were much more extensive. The implementation team comprised of a cross-functional team including managers, several nurses, doctors’ assistants from the triage department and outpatient clinic, at least two professionals of the medical team, and an IT professional for adding the digital application of the DOTTS into the electronic patient record system. The implementation team jointly developed a tailored implementation plan, which was an actionable specific work plan for the users of each individual hospital. This work plan included the following steps: researching whether there is support for the innovation, performing a baseline measurement, and formulating relevant goals of triage. In addition to the formulated goals, it was important to organize the right facilities, such as a physical workplace for obstetric triage with a computer, telephone, and headset. In addition, an important step during the implementation process was integrating the digital application of the DOTTS into the hospital’s electronic patient record system. Importantly, in all hospitals, specific training about the DOTTS was given to the staff responsible for triage. In most hospitals, this training was outsourced to an external organization. In some hospitals, this training was given by in-house experts. This choice was determined by the implementation team. Lastly, providing information to third parties before and after implementation was also an important step. The information about the innovation was given to patients, colleagues, and other cooperation partners (eg, general practitioners and the hospital emergency department).

The implementation team went through the implementation strategies before getting started with the DOTTS. The order, as well as the extent of the steps performed, differed per hospital. Progress was evaluated and adjusted through interim process evaluation. The steps were not taken sequentially. Most implementation teams used process steps, which, in retrospect, show similarities with the process models of Kotter or Grol and Wensing [23-25]. These frameworks are intended to support stepwise planning and management of implementation efforts.

The participants of this study were users of the DOTTS. Users of the DOTTS were obstetrical nurses, nurses, and doctors’ assistants, and they can be seen as triage staff. Users from all hospitals where the DOTTS was implemented were included. Before inviting participants for the questionnaire survey and the focus group discussion, an exploratory meeting was held with the manager of the department where the participants were employed. In this meeting, the manner of invitation of participants was discussed. Hence, a list of participants was formed. According to the preference of the managers and the intended participants, either email addresses were provided or an information letter including a hyperlink to the questionnaire was forwarded to the participants by the manager. At the end of the questionnaire, the participants were asked if they also wanted to participate in a follow-up study (focus group). After consent, these participants were contacted again for participation in the focus group.

All participants were informed about the study and provided digital informed consent prior to the use of the data for analysis. All data were anonymously processed. Participants were able to withdraw at any time, without any statement of reason. The study was approved by the boards of the Medical Research Ethics Committees United, the Medical Ethics Committee of Leiden University Medical Center, and Erasmus MC of Rotterdam (W.16.053 & P17.075/PG/pg & C1.20191125).

In total, 294 triage staff members from the 9 hospitals were asked to complete the questionnaire. The overall response rate, after 3 reminders, for complete responses was 58.8% (173/294).

The participants who filled out the questionnaire had a mean age of 43.3 years (SD 11.6 years) and an average work experience in obstetrics of 17.9 years (SD 11.5 years). Participants in the focus group had a mean age of 46 years (SD 9.2 years) and an average work experience in obstetrics of 18.6 years (SD 9.9 years). An overview of the characteristics of the participants is provided in Table 1. In total, 156 of the 173 (90.2%) participants had used the DOTTS for over 6 months. The digital application of the DOTTS was used “as much as possible” or “always” by 137 of the 173 (79.2%) participants.

The overall NPS score was 3.77 (SD 0.36). The constructs coherence, cognitive participation, collective action, and reflexive monitoring scored 4.01 (SD 0.47), 4.05 (SD 0.45), 3.5 (SD 0.45), and 3.72 (SD 0.47), respectively (Table 2). On average, all participants agreed (score 4) with the statements associated with the constructs coherence and cognitive participation. For the constructs collective action and reflexive monitoring, the scores were between 4 (agree) and 3 (disagree). These results were also seen when each hospital was analyzed separately (Multimedia Appendix 2). The scores for the constructs collective action and reflexive monitoring showed more variation compared to the scores for the constructs coherence and cognitive participation (Figure 1). All elements were recognized by most participants. The constructs coherence and cognitive participation had a high percentage of answers with “agree” and “strongly agree” (Multimedia Appendix 3). In the pooled data set, Cronbach α was .85 for the total NPS score and was .71 for coherence, .70 for cognitive participation, .67 for collective action, and .68 for reflexive monitoring (Table 2).

Eight participants (Table 1) discussed the implementation of the DOTTS in their hospitals and what lessons could be learned. The focus group discussion lasted 90 minutes. The focus group was highly valued by the participants and experienced as a reflection moment. Participants discussed whether they agreed with the NPS and construct scores, and came up with possible explanations.

This study aimed to evaluate the use of the DOTTS in daily practice after its implementation in a hospital. Evaluation focused on daily use and on the 4 constructs of the NPS. The DOTTS was used by almost all participants over a period of 6 months or more. The digital application of the DOTTS was used as much as possible or always by most participants. The overall score of the NoMAD questionnaire was 3.77 (SD 0.36). There were some differences per construct, where coherence and cognitive participation scored better and with less variation than collective action and reflexive monitoring. Outcomes of the focus group discussion confirmed the added value of the DOTTS. Use was stimulated by the presence of a dedicated multidisciplinary team and supported by medical staff, as well as proper use of the triage ward, adequate training, and official evaluation.

Our results are in line with and, in some aspects, better than the results of other evaluation studies with complex implementations using the NoMAD questionnaire [26,34,35]. While the Dutch questionnaire was previously applied to e-mental health interventions [26], this is the first time it was applied in obstetrics. To assess whether the reliability in a different area of health care is sufficient, we also looked at the Cronbach α. The results of Cronbach α showed good internal consistency for the total NPS score and acceptable findings for coherence and cognitive participation. However, the results were questionable for the constructs collective action and reflexive monitoring. Our findings are comparable to previous results when using the Dutch version of the questionnaire [26].

Triangulation of our results was facilitated via a focus group discussion. What emerged from this discourse was that triage professionals were able to see the added value (coherence) and were committed (cognitive participation), but struggled with collaboration (collective action) to use the DOTTS and did not always reflect on their efforts (reflexive monitoring) in a systematic manner. A plausible explanation for the favorable results of coherence and cognitive participation is the early and intensive involvement of stakeholders in the development and implementation of the DOTTS, which supports implementation. Stakeholders were involved in the development and gave their commitment about the use of the DOTTS in daily practice. The innovation was created together and is therefore well suited to the needs of care providers [5,36-38]. The construct collective action showed the widest variation. One possible explanation for this variation is the tailor-made approach adopted for the implementation plan. The order, as well as the extent of the steps of the implementation plan, differed per hospital. Moreover, the context differed per hospital, which means that every implementation was also different [12].

The lesson learned from this study is that evaluation (ie, reflection) of preplanned, systematic, and strategic implementation of an innovation in health care deserves more attention. In our study, we mainly evaluated whether the use of the DOTTS normalized after implementation (ie, a state of affairs). Each hospital made a tailor-made plan per implementation, which retrospectively showed similarities with the process models of Kotter or Grol and Wensing [6,23-25]. The change management model by Kotter [25] and the model by Grol and Wensing [6] intended to support the planning and managing implementation efforts. In this study, there was no specific model used; therefore, it is difficult to compare the results with these well-known implementation models. However, evaluation of use, similar to the construct reflexive monitoring, is also an important step in these models.

Improving the quality of services for pregnant women was seen as important after the implementation of the DOTTS. Improvement of quality is mentioned in most implementation science research as a condition for success [14]. Improper use of the triage ward, which was considered by participants as a barrier, is also a well-known phenomenon in the organization of care within the hospital and specifically in triage wards [38-40]. Commonly, in planned hospital care, capacity is limited, causing nonurgent care events to be diverted to the emergency care department. Several factors leading to improper use were also mentioned in the scoping review of Bailey et al [41]. Improper use brings challenges, such as workload stress, which subsequently influence decisions during telephone triage [40,41].

In line with the constructs collective action, cognitive participation, and coherence, implementation by a dedicated multidisciplinary implementation team, which provides guidance during implementation and use of the tool afterwards, was mentioned as important. In our study, this referred to the importance of the involvement of all stakeholders. Preparing an innovation with all stakeholders creates the possibility of optimal support from the start and user friendliness for all stakeholders in daily practice [5,14,42]. Within the multidisciplinary team, special attention should be paid to the participation of the medical group. To change medical staff routines, leadership of the implementation team is an important element [43]. Hierarchy is also a challenging factor here, which is in line with results from other studies indicating that in a hierarchical organization, normalization of an innovation is often more difficult [35]. If the organization of care is arranged by the nurse, it is necessary that it is supported by medical staff [40].

We found that training, which is part of the construct collective action, is an important element of change. This was also seen by existing triage systems [38,44,45]. If users themselves have a need for training because they want to be competent, this contributes to success [46]. After completing the training, it is important to provide continuous evaluation, which contributes to the construct reflexive monitoring, so that the implementation is further optimized and users receive confirmation that they are doing well [38].

The NPT with its validated NoMAD questionnaire was used as an evaluation framework in this study. The NPT is an implementation theory and has been widely used as an evaluation framework [11]. There are also other tools that reflect the success of implementation, such as the Consolidated Framework for Implementation Research (CFIR); Nonadoption Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework; and Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework [11]. The NPT was used in this study because the context of the implementation of an innovation in obstetric care corresponded to the NPT [15].

The use of 2 complimentary research methods is valuable in the interpretation of data. A large group of users were given the opportunity to evaluate the use of the DOTTS, providing a general overview of implementation in several hospitals. The focus group gave the opportunity to triangulate the outcomes, thereby gaining more insight into the meaning of the answers and clarifying the context.

The results are from all hospitals that implemented the DOTTS before September 2019. There was an overall response rate of 58.8% (173/294) from these 9 hospitals. A 50% response rate, which was obtained from all hospitals, can be considered representative [47] (Multimedia Appendix 2). With an average age of 43.3 years and experience of 17.9 years, the sample composition was representative compared to other studies within this profession [48-50]. The participants of the focus group showed good representation of the total research group (Table 1).

To improve the questionnaire, it is recommended to look to the question collective action-2 of the construct collective action because it is the only negatively asked question. It is unclear if every participant interpreted the question correctly.

In this study, we chose to evaluate the degree of normalization after implementation among daily users in 9 hospitals where the DOTTS is offered as usual care. This focus resulted in a lack of information about the theoretical approaches used for each implementation strategy. The tailored implementation strategy created space for context per hospital and the team of stakeholders. However, it limited the ability to evaluate effectiveness per implementation strategy. In addition, due to the current aim of this study, results on the expected quality improvement were lacking. Furthermore, this study only looked at the perspectives of the daily users of the DOTTS. The lack of perspectives of medical staff, outpatient clinic staff, management, and other related professionals is a potential limitation. In view of the importance of tailored implementation strategies, which was highlighted by our research, we recommend that a future study should include representation from the medical group to ensure an inclusive perspective.

Moreover, we did not evaluate the patient perspective. Not every patient will fit into the evidence-based system of the DOTTS, for instance, patients who do not understand self-care with advice. Customization per patient might need to be further developed. The current evaluation was not about these items and requires further research. Further insight into the experiences of patients who have received telephone and physical triage care based on the DOTTS is therefore recommended.

Normalization of the DOTTS was seen after tailored implementation in 9 hospitals. Key factors in the normalization process of the DOTTS in obstetric triage were (1) the shared added value for stakeholders; (2) the dedication of the complete multidisciplinary implementation team with specific support from medical staff, as well as proper use of the triage ward (as designed) by all disciplines; and (3) implementation plans that are tailor made in the practical context of the hospital. Improvement can be achieved by structuring this process and incorporating implementation strategies, such as systematic training and evaluation, with users.