This feasibility study used a quasi-experimental pre- and posttest design in which trained simulated callers conducted standardized calls to operators in medical communication centers before and during the use of the prototype. The primary outcomes were overall medical quality and overall communication quality. Secondary outcomes for medical quality included performance on specific medical content items, accuracy of triage, and patient safety, whereas secondary outcomes for communication quality included performance on specific communication items, call duration, and efficiency.

In Norway, primary and specialist health care are legally required to provide 24/7 emergency medical services [30]. Emergency primary care, run by the municipalities, is responsible for the population’s access to emergency primary care clinics and local emergency medical communication centers (LEMCs), while specialist health care, run by the state, is responsible for the ambulance service and the ambulance dispatch centers [31]. Through the European number 116117, the LEMCs serve as a gateway to acute and urgent health care by prioritizing patient situations by urgency, initiating responses, and providing medical advice [30]. The LEMCs handle medical calls covering contact reasons ranging from emergencies to nonurgent medical issues, as opposed to ambulance dispatch centers that primarily handle medical emergencies.

The study was conducted at 5 LEMCs in Norway, all of which volunteered to participate after the study objectives had been presented at a national conference for managers in emergency primary care services in March 2022. Each LEMC independently recruited 5 operators. To enable before-after comparisons, the same operators were required to take part in both simulation rounds. Therefore, they had to be fully informed about the study requirements and motivated to complete both phases. To support operator recruitment, an informational video describing the study was shown during staff meetings or distributed via email. Operators willing to participate reported their interest to their manager or a designated contact person at the emergency primary care clinic.

For nonprofessional reasons, 3 operators opted out of the posttest and were therefore excluded from the study. The 22 remaining operators were female registered nurses aged 27-58 years. Among them, 10 operators had 5 years or less of work experience in an LEMC, while 12 had more than 5 years of experience.

The simulated callers were 4 women and 1 man with limited acting experience. They were recruited and trained by an actress who specialized in training simulated patients [32]. One of the simulants withdrew after the pretest and was replaced by a new participant of the same gender and age. Training included 1 individual and 2 group sessions, focusing on the character and context of each case. To ensure medical accuracy, 2 researchers (SLSF and IHJ) with experience in telephone triage served as operators during the training sessions.

Fifteen medical cases representing a range of conditions were developed by the project team. An overview of the cases is provided in Multimedia Appendix 1. Each case described the patient’s condition, contextual details, and specific characteristics of the caller’s role. Without being a fixed script, the case description specified the opening lines that should be provided by the caller at the beginning of the call and information that could be provided later in response to the operator’s questions.

The prototype was developed as part of the research and innovation project “RE-AIMED: Readjusted Responses by Use of AI in Medical Calls.” Several operators from various LEMCs were involved in the development process. This process is described in detail by Berge et al [5].

The prototype’s user interface contained five different components (Figure 1):

After the pretest, all operators were introduced to the prototype. They had the opportunity to use it from mid-November 2022 until the completion of the posttest in mid-March 2023. Throughout this period, the operators tested the prototype on anonymous conversations recalled from their own practice. These inputs were used as additional training data for the ALS and RNN models. The operators provided feedback on errors and suggested improvements to the prototype. Follow-up meetings, either digitally or in person, were arranged. Before the posttest, operators used the prototype between 2 and 191 sessions, with a mean of 51 sessions. To support case-relevant question suggestions during the posttest, the recommender system was additionally trained on the 15 simulated cases.

The primary outcomes were overall medical quality and overall communication quality, both measured as continuous variables on a 0-10 scale. Secondary outcomes related to medical quality were assessed using (1) medical content items: 9 categorical items with 5 response categories, (2) triage accuracy: a categorical variable with 7 levels, and (3) patient safety: measured on a continuous 0-10 scale. Secondary outcomes related to communication quality were assessed using (1) communication items: 9 categorical items with 5 response categories, (2) call duration: measured in seconds (continuous variable), and (3) efficiency: measured on a continuous 0-10 scale.

The unit of analysis was the individual call. Linear mixed‑effects models were used to compare pretest and posttest scores for the continuous outcomes. Operator and case identifiers were included as random intercepts to account for clustering at both levels. Variance components were used to calculate intraclass correlation coefficients (ICCs) for operators and cases, which were interpreted according to established guidelines [34]. Results are presented with means, mean differences, 95% CIs, corresponding P values, and ICCs. Effect sizes for paired comparisons (Cohen dz) were calculated by dividing the mean of the paired differences by the standard deviation of those differences [35]. Effect sizes were interpreted according to conventional benchmarks, with values around 0.2 considered small, 0.5 medium, and 0.8 large.

For categorical outcomes, changes between pretest and posttest scores were illustrated descriptively using 100% stacked bar charts. An exception was triage accuracy, which was presented in a 7×7 transition matrix with frequencies. Statistical changes in categorical outcomes were analyzed using Bowker test for table symmetry, with results reported as P values. The score category “NA” was used when the item was correctly omitted in a call. Since this omission represented correct performance, this value was recoded to the item’s highest scoring category. A sensitivity analysis was conducted by repeating Bowker test with NA items excluded. This allowed us to assess whether the findings were robust to alternative treatments of NA responses. Statistically significant categorical results were visualized using bubble plots, where bubble size represents the number of observations and the position reflects transitions from pre‑ to posttest scores. A significance level of α=.05 was used for primary outcomes. To account for multiple comparisons, a Bonferroni‑adjusted significance threshold of α=.002 was applied to the secondary outcomes. All analyses were performed using Stata/SE (version 19; StataCorp) [36].

The study did not fall under the Norwegian Health Research Act, and the Regional Committee for Medical and Health Research Ethics waived the requirement for ethical approval (reference number 227587). The Norwegian Social Science Data Services (reference number 133364) and the Data Protection Officer for Research at NORCE Research AS approved the processing of personal data, including audio recordings and operator documentation. All participants, including operators and the simulated callers, provided written informed consent for participation and for the use of their data for research purposes. Study data were anonymized before analysis, and procedures for handling, storing, and securing data adhered to General Data Protection Regulation and national data protection regulations. Only authorized members of the research team had access to identifiable information. The paper contains no images or supplementary materials that could identify individual participants. Operators received their regular salary from their respective LEMCs for the days they participated in the simulation, while the LEMCs were compensated with NOK 1500 (US $158) per operator per day. Simulated callers were compensated through the performing arts company that provided their services.

Of 330 planned paired calls, 10 were excluded due to unavailability of the caller or the operator at the scheduled call time (n=4) or failure to retrieve the audio recordings (n=6). Thus, 320 paired calls remained for analysis. Participant flow through the pretest and posttest simulation sessions, including exclusions and the final number of paired calls analyzed, is shown in Figure 2.

Overall medical quality improved significantly (difference 0.34, 95% CI 0.11-0.57) (Table 2). Variance components estimated showed an ICC of 0.17 for operators and 0.42 for cases. The effect size was small (Cohen dz=0.16). No statistically significant difference was observed in overall communication quality between pretest and posttest (Table 2). The corresponding effect size was very small (Cohen dz=0.05).

Figure 3 illustrates how the scores within each specific medical item were distributed between pre- and posttest. In 16%-43% of the calls, specific items were incorrectly omitted or insufficiently managed.

Paired comparisons showed a statistically significant decrease for the item “Collects information about the patient’s location” (P<.001) (Figure 4). Otherwise, no statistically significant differences were observed. The sensitivity analysis yielded P values comparable with the main analysis (Multimedia Appendix 3), indicating that the results were robust to the treatment of NA responses.

For triage accuracy, 69% (222/320) of cases were optimally triaged in the pretest and 75% (240/320) in the posttest (Table 3). Triage accuracy did not differ significantly between pre-and posttest (P=.30). There was also no change between pre- and posttest in overall patient safety, with a mean score of 7.92 in the pretest and 7.99 in the posttest (difference 0.07, 95% CI –0.14 to 0.27; P=.53). ICC was 0.14 for operators and 0.41 for cases, and the effect size was small (Cohen dz=0.04).

Figure 5 illustrates how the scores within each specific communication item were distributed between pre- and posttest. In 1%-27% of the calls, the items were incorrectly omitted or insufficiently managed.

Among the communication items, a statistically significant difference was observed in the item “Ensures that the triage decision and the advice given are understandable and feasible” (P=.002), with higher scores in the pretest (Figure 6). Consistent with the sensitivity analysis for the medical content items, the P values were comparable with those of the main analysis (Multimedia Appendix 3).

There was no statistically significant change in mean call duration, which was 317 seconds in the pretest and 308 seconds in the posttest (difference –9 seconds, 95% CI –20 to 1 seconds; P=.08). ICC was 0.30 for operators and 0.72 for cases, and the effect size was small (Cohen dz=0.10). Overall efficiency showed no change from pretest (mean score 6.67) to posttest (mean score 6.84) (difference 0.17, 95% CI –0.08 to 0.41; P=.19). ICC was 0.18 for operators and 0.36 for cases, and the effect size was very small (Cohen dz=0.07).

The results show that use of a prototype, designed to dynamically suggest medical questions and provide structured documentation, influenced the operators’ interactions with the callers in a scenario-limited environment. Overall medical quality improved significantly when the prototype was used, albeit with a small effect size. Secondary outcomes showed a decrease in the items “Collects information about the patient’s location” and “Ensures that the triage decision and the advice given are understandable and feasible.” The remaining overall quality items, specific items, and call duration showed no significant change between the pre- and posttest.

To our knowledge, no previous studies have examined the use of recommender systems during telephone triage. There seems to be a gap in the current evidence base for how health recommender systems lead to behavioral or performance changes, particularly among health professionals [20,21,24,37]. One exception is a study evaluating clinicians’ use of a machine learning–based recommender system during simulated order entry to initiate a component of patient care [38]. Although the recommender did not improve the clinical appropriateness of orders, it was extensively used, perceived as useful by most participants, and associated with a modest increase in ordering behavior. Previous work shows that recommender systems can reduce information overload and provide more targeted, context‑relevant support [21]. One example is the use of AI‑based recommenders in a suicide prevention helpline, which generated real‑time suggestions that were especially used during complex or longer calls [39]. Altogether, this suggests that recommender systems may be useful in health professionals’ work, even when improvements in clinical appropriateness remain limited.

The use of the prototype provided an improvement in overall medical quality, operationalized as recognition and prioritization of symptoms alongside the delivery of relevant medical information. The improvement was not reflected in any of the specific items related to medical content, providing no indication of which aspects of the medical content that changed. In addition, the effect size was small, and no change was observed in triage accuracy or patient safety, indicating that the clinical relevance of the change is limited. It is possible that the prototype’s full potential was not captured in this study as operators already demonstrated high performance during the pretest. This was especially relevant for the overall quality item patient safety, which showed a score of 7.92 out of 10 in the pretest. Such high baseline scores create a ceiling effect, leaving limited room for observable improvement and making small performance changes difficult to detect, even when actual performance may have improved.

The functionalities of the prototype were not tested in isolation, and we cannot determine which specific feature contributed to the observed change. The AI-based question recommender functionality may have contributed to the improvement in medical quality by dynamically presenting context-relevant questions. This may have acted as a subtle nudge, guiding operators toward more comprehensive information gathering. Nudging has been shown in other clinical contexts to influence clinician behavior by drawing attention to relevant cues and thereby enhancing decision‑making [40,41]. In a review, Jesse and Jannach [25] have suggested that a substantial potential of recommender systems lies in their ability to implement digital nudging that supports users in making more informed choices.

Structured documentation has been described by nurses to aid their memory and overview during consultations [29], and conversation‑analytic research demonstrates that structural resources, such as agendas or templates, shape how clinicians organize and navigate interactions [42]. The second functionality of the prototype, structured documentation with high urgency symptom combinations highlighted, dynamically presented the confirmed or denied questions organized according to a predefined hierarchy. This may have helped the operators to keep track of information already covered and to identify remaining or critical information gaps, thus improving the overall medical quality.

Noteworthy, the specific medical item “Collects information about the patient’s location” showed a statistically significant decrease from pretest to posttest. As the instructions given to the operators on how to handle information on the patient’s location was similar before both test rounds, it is unlikely that the decrease can be attributed solely to the simulated setting. The operators may have been influenced by functionalities in the prototype such as questions concerning patient location not being included in the prototype’s question set, and location not being represented within the user interface alongside the standard demographic items (eg, age or sex). Prior research shows that while prompts and nudges can enhance certain behaviors, they may also unintentionally redirect attention away from nonprompted but still essential tasks [43]. This underscores the need to monitor not only the positive effects of such systems but also possible unintended negative consequences for operator performance.

No change in overall communication quality was observed, which is noteworthy given prior concerns that decision support systems might impair the natural communication flow between the operator and the caller [13-15]. Call duration and overall efficiency also remained unchanged. However, when examining specific communication behaviors, fewer operators asked callers to confirm their understanding of the triage decision and advice in the posttest. This suggests that, although the prototype did not reduce communication quality at a global level, it may have subtly shifted the interactional dynamics between operators and callers. Communicative items are typically not embedded in decision support systems but rely on the operators’ adherence to established communication practices [44]. Recent research indicates that AI‑based systems can provide stable and supportive conversational cues, suggesting that integrating communication guidance may enhance both user experience and the thoroughness of information gathering [45]. Given the importance of both accurate information gathering and effective communication for high‑quality telephone triage, future prototypes may benefit from integrating nudges that target both medical content and communicative behaviors.

This study relied on a compound prototype and simulated callers to ensure a controlled comparison between the pretest and posttest conditions, which allowed us to compare the operators’ performance over similar cases in both rounds. However, the study design had several limitations.

First and foremost, the prototype included several components that could influence the operators’ performance. Furthermore, a simulated environment is limited in its ability to capture the variability and unpredictability that characterize real-world calls. These factors limit how the findings can be used in the design of decision support systems for clinical settings.

Second, the recommender system was overfitted by being additionally trained on the very scenarios used in the posttest to generate case‑relevant question suggestions. The amount of relevant training data was limited. By training on the test scenarios, we aimed to test the behavior of a well-performing recommender system. Consequently, the observed improvement cannot be extended to real‑world cases beyond the simulated scenarios.

Third, the use of a single rater introduced a potential risk of researcher bias. To mitigate this, the conversations were assessed using a validated rating tool with dual scoring in the initial phase, and any uncertainties were discussed within a subgroup of the author team throughout the scoring period. Furthermore, calls were scored in random order, and the rater was blinded to the test period, location, and operator identity for each call. Consequently, any remaining rater bias would likely have influenced both test rounds in similar ways.

Fourth, there is a risk of carryover effects because the operators handled the same case twice. Although operators were instructed not to discuss cases with colleagues, and the 2 test rounds were separated by a 5-month interval during which they were estimated to have handled approximately 780-1410 unrelated calls [46], recall bias cannot be ruled out. The long interval and the high volume of intervening calls likely reduced this risk. However, the 5‑month gap also introduces a potential maturation effect, as especially less experienced operators may have naturally improved their professional performance over this period, independent of the intervention.

Fifth, operator recruitment was based on voluntary participation. This may have introduced motivation bias, as more engaged or higher‑performing operators may have been overrepresented among the volunteers. Such a skewed sample may have contributed to the high baseline scores observed, potentially resulting in a ceiling effect and leaving limited room for measurable improvement. The high baseline scores may also be partly explained by the simulated setting, in which the simulated callers used a predefined opening line that stated the purpose of the call. This likely simplified the task compared with clinical reality, where the purpose of the call might be less explicitly stated, potentially inflating overall quality scores.

Sixth, the sample size was set in accordance with the project’s financial, temporal, and personnel constraints, which led to a relatively small sample that may have restricted our ability to conclude. Also, the requirement to use the same operators in both rounds made the study design vulnerable to dropouts. Despite strong motivation to participate, 3 operators withdrew before the second round, reducing an already limited sample available for comparisons and thereby further constraining the ability to detect potential effects of the prototype.

The findings of this study indicate that a prototype combining recommender functionality with structured documentation can improve medical quality. It seems reasonable to further explore how such functionalities can be developed and incorporated into a full decision support system to ensure both medically accurate support and sustained high‑quality communication during telephone triage. Further studies are needed to understand the contributions of each component of the prototype and to validate the usefulness of question recommenders in settings that have a huge variety in reasons for contact. Operator recruitment should be designed to minimize the risk of ceiling effects by ensuring a more diverse sample of operators, either through broader recruitment or through more randomized selection.

Since the RE-AIMED project started in 2020, advances in AI have accelerated rapidly, with large language models (LLMs) becoming increasingly powerful and accessible. This progress has expanded the use of natural language processing. Compared with traditional models relying solely on structured data, natural language processing–based approaches have been shown to improve classification performance when unstructured free text is incorporated [47]. This opens the possibility for more seamless collaboration between the operator and the machine, where recommendations and documentation are generated directly from the model’s analysis of the ongoing conversation rather than only reacting to the operators’ inputs. Recent work demonstrates that LLMs can provide consistent, context‑sensitive suggestions that enhance both emotional support and information gathering during safety‑related conversations [45], suggesting that LLM‑based systems may likewise assist telephone triage operators by providing real‑time prompts that support empathy, reduce information overload, and improve the overall quality of operator-caller interactions.

The use of a prototype that combined AI‑based question recommendations with structured documentation yielded a modest improvement in overall medical quality within a scenario‑limited environment. While overall communication quality remained unchanged, aspects of the interaction were negatively affected. It appears feasible that AI‑based question recommendations and structured documentation may serve as useful functionalities within a decision support system. However, these functionalities require further development and evaluation before being used in clinical settings.