The privacy-preserving system contains software components that are deployed at each health institution. The system extracts data daily from the local EHR system and transforms and loads the data into a database that conforms to a common data model [14].

A third party (denoted as a coordinator) aids the system without learning private information and is trusted to follow protocol specifications. This is a standard security model known as the honest-but-curious adversarial model [15]. In this study, the Norwegian Centre for E-health Research acted as the coordinator. The coordinator accepts queries for the aggregated performance indicators of GPs across health care institutions. The system uses privacy-preserving distributed data mining techniques [16-18] for executing statistical queries on the combined data of health care institutions without allowing any party to view the private data that health care institutions compute locally. Statistical results on the combined data of multiple health care institutions are not considered sensitive information and are therefore stored at the coordinator and retrieved through a web service.

The feedback report is generated locally by combining local personal indicators and the aggregated indicators retrieved from the coordinator. Access to the feedback report is restricted to the respective GP and is provided through a web client or email as an encrypted PDF file, and the decryption key is sent to a mobile phone. We refer interested readers to a study by Yigzaw et al [13] for an elaborated description of the system.

In 2019, we deployed the system in three GP offices in Norway, and 20 GPs received a single feedback report on their prescribing of antibiotics for selected RTIs. The GPs were then asked to fill out a web-based questionnaire for assessing their perceptions of the feedback received and how the feedback was presented.

Norwegian GPs are responsible for treating a set of patients and refer only those who need more specialized health services to hospitals [19]. Therefore, the 20 GPs had a total of around 19,345 patients on their lists. The number of registered patients varied during the study period because patients sometimes change their GP.

The GPs received a single feedback report on their prescribing of antibiotics for selected RTIs for which antibiotics are generally not recommended if the patient is otherwise healthy (eg, during the first presentation of a current episode of an RTI or when the patient has no significant underlying comorbidity) [20]. Based on the International Classification of Primary Care, Second Edition [21], the selected RTIs were acute upper respiratory infection (R74), acute sinusitis (R75), acute laryngitis/tracheitis (R77), acute otitis media/myringitis (H71), and unspecified respiratory infections (R83). Acute bronchitis (R78) was also included because it is most often a viral infection, and the use of an antibiotic is rarely recommended.

Feedback was provided for the selected RTI cases between 2015 and 2018; both combined and separate feedback were provided for each of the diagnoses. All statistics were stratified by both year and antimicrobial spectrum (broad- and narrow-spectrum antibiotics). Based on the Anatomical Therapeutic Chemical classification system [22], narrow-spectrum antibiotics include β-lactamase–sensitive penicillins (J01CE), and broad-spectrum antibiotics include tetracyclines (J01A); β-lactam antibacterials and penicillins (J01C, excluding J01CE); other β-lactam antibacterials (J01D); sulfonamides and trimethoprim (J01E); macrolides, lincosamides, and streptogramins (J01F); and quinolone antibacterials (J01M).

The feedback report contains the number of cases that were diagnosed and treated with antibiotics. A set of performance indicators was presented as a time series graph that compares the performance indicators of a GP with the average indicators of all participating GPs. Performance indicators were based on the indicators proposed by the European Surveillance of Antimicrobial Consumption Network [23], as follows:

A panel of experts with clinical and medical informatics backgrounds developed the questionnaire. As shown in Multimedia Appendix 1, the questionnaire broadly covers aspects that were derived from the following three theoretical constructs of the Theory of Planned Behavior [24]: (1) attitude toward privacy and the prescribing of antibiotics, (2) subjective norms (reflection on the accuracy and appropriateness of using feedback as the basis for comparisons with peers), and (3) self-efficacy regarding changing prescribing behavior.

Descriptive statistics were calculated to summarize the results of the categorical questions, including age groups and years of experience.

This study was based on an anonymous questionnaire and therefore did not require ethics review board approval, according to Norwegian regulations. However, all participants gave their informed consent to participate in this study.

The retrospective data that were generated by using the A&F system show that approximately 20% (2924/14,396, 20.3%) of all cases of the selected RTIs were treated with antibiotics, as illustrated in Figure 1.

A total of 14 GPs responded to the questionnaire, representing a 70% (14/20) response rate. Of these GPs, 36% (5/14) were aged 50 years or older, 57% (8/14) had more than 15 years of experience as a GP, and 79% (11/14) had a specialization in family medicine. The survey results on attitudes toward privacy, audits, and feedback are summarized in Table 1.

We can observe from Table 1 that the GPs were unanimous about privacy protection for their patients, and 64% (9/14) of GPs considered the protection of their own privacy to be very useful or useful.

Most of the GPs (13/14, 92%) reported that feedback on their prescribing of antibiotics and the ability to compare their performance indicators with those of their peers were useful or very useful. The clinicians preferred to receive feedback at regular intervals, such as half-yearly (5/13, 38%) and yearly (7/13, 53%; 1 response was invalid). Most of the clinicians (13/14, 92%) preferred to receive an encrypted feedback report through email rather than having it integrated into the EHR (3/14, 21%), receiving it on a secure website (1/14, 7%), or receiving it on a mobile app (0/14, 0%).

Of the 14 GPs, 1 (7%) indicated the intention to change their antibiotic prescribing behavior based on the feedback received, 2 (14%) indicated that they did not intend to change, and the rest (n=11, 78%) were unsure of whether they would change.

Most of the GPs (11/14, 78%) wanted their feedback to be stratified by the characteristics of their patient populations and diagnoses. Age and gender were the commonly requested patient characteristics for stratification. The clinicians also requested stratification by chronic conditions, such as chronic obstructive pulmonary disease, asthma, cystic fibrosis, other airway comorbidities, acne, perioral dermatitis, and heart failure.

Perhaps the major highlight of this study relates to privacy preservation, since the A&F system resolves most of the privacy concerns raised in similar studies. Our results show that the GPs value the protection of their own privacy as well as that of their patients, and this finding is in line with those of existing studies [25]. Our results appear to support our initial assumption that protecting the privacy of clinicians may increase their willingness to participate in quality improvement activities.

GPs in Norway often work in small practices with few or no peers; therefore, comparisons with peers across multiple institutions could be especially useful, and the privacy safeguards serve as further incentives for participation.

Our results show that the GPs were in favor of receiving encrypted feedback reports through their emails. This contrasts with the common belief that integrating a clinical decision support system with EHR systems is an important success factor [26]. It is plausible that the GPs assumed that integration with the EHR would trigger distractive feedback alerts more frequently than those triggered by the desired half-yearly or yearly periodic feedback.

The GPs appeared to value both seeing their antibiotic prescribing statistics and being able to compare these with those of their peers. Although this study did not measure whether the clinicians actually changed their prescribing behavior following feedback, it is likely that most of the participants had a close-to-average prescribing practice and would therefore not have an incentive to change their prescribing behavior. In a larger sample of clinicians, there would be a higher number of individuals that deviate from the mean and have an incentive to change their prescribing behavior.

The GPs requested the stratification of feedback by patient characteristics, such as demographics and chronic diseases, of which both are known to influence the decision to prescribe antibiotics. Stratification provides important information to clinicians on whether their prescribing behavior can be justified based on patient characteristics and enables peer comparisons among clinicians with similar patient populations.

An important limitation is that we could not evaluate GPs’ responses based on their clinical performance, since the questionnaire was anonymous and access to feedback reports was limited to the respective clinician. For example, we were not able to assess how the self-reported intention to change related to an individual’s current clinical performance.

Another limitation is that the feedback reports provided to GPs were not adjusted for comorbidities and age. These two variables are known to influence the decision to prescribe antibiotics, and this may have affected their responses to the survey.

We also noted that the average performance of peers might be skewed if there are outlier clinicians who are far from the mean on 1 side. Therefore, it might be necessary to exclude outliers or provide more statistics, such as SDs and percentiles, to enable detailed comparisons with peers.

The sample size in this study can be considered small but can also be viewed as appropriate for an in-depth case analysis for a pilot implementation of this type.

The findings from this study have important implications for practice, especially implications related to quality improvement programs at health care institutions. We single out 3 elements that we recommend for improving systems that support antibiotic stewardship programs. The first and basic element is protecting the privacy of clinicians as well as that of patients. Second, scalability should be considered so as to enable comparisons among peers across multiple health care providers. Comparisons across health care providers are especially useful for small practices like GP offices, since such comparisons require a wide pool of similar patient groups and can be adjusted for comorbidities and age. Finally, human factors should be considered. For example, the frequency of feedback should be considered, since clinicians may not want distractive feedback alerts.

In terms of impact, our system offers a solution to key challenges that hamper antibiotic stewardship programs. It provides privacy guarantees to patients, clinicians, and health care institutions and provides the scalability required to ensure that long-term audits and feedback can be sustainable parts of stewardship programs.

The impact for Norway might be limited, since Norway is a small country with relatively lower antibiotic prescribing rates compared to those of other European countries [27]. For larger countries with differently organized general practices, a higher effect can be expected, since audits and feedback are the most effective in places where the baseline performance is poorer compared to that of the best general practices [7].

Our results show generally positive sentiments among GPs regarding the potential impact of our system, but it should be noted that this was an initial study with a relatively small number of participants. Therefore, the absolute number of clinicians with very high prescribing rates was likely to be low. In larger samples, we expect that a higher number of physicians would receive an incentive to change their practice.