Background: Antibiotic resistance is a worldwide public health problem that is accelerated by the misuse and overuse of antibiotics. Studies have shown that audits and feedback enable clinicians to compare their personal clinical performance with that of their peers and are effective in reducing the inappropriate prescribing of antibiotics. However, privacy concerns make audits and feedback hard to implement in clinical settings. To solve this problem, we developed a privacy-preserving audit and feedback (A&F) system.
Objective: This study aims to evaluate a privacy-preserving A&F system in clinical settings.
Methods: A privacy-preserving A&F system was deployed at three primary care practices in Norway to generate feedback for 20 general practitioners (GPs) on their prescribing of antibiotics for selected respiratory tract infections. The GPs were asked to participate in a survey shortly after using the system.
Results: A total of 14 GPs responded to the questionnaire, representing a 70% (14/20) response rate. The participants were generally satisfied with the usefulness of the feedback and the comparisons with peers, as well as the protection of privacy. The majority of the GPs (9/14, 64%) valued the protection of their own privacy as well as that of their patients.
Conclusions: The system overcomes important privacy and scaling challenges that are commonly associated with the secondary use of electronic health record data and has the potential to improve antibiotic prescribing behavior; however, further study is required to assess its actual effect.
Antibiotic stewardship is an issue that has recently come under scrutiny in Norway  and internationally [ ], especially in general practices, which is where most antibiotic prescribing occurs [ ]. The inappropriate use of antibiotics is thought to be an important factor in the development of antimicrobial resistance. Although a number of guidelines define when antibiotic use is warranted, defining appropriate use is difficult, unless there is clear agreement on the etiology of an infection [ ]. There are diverse and complex reasons for the overuse of antibiotics, and consequently, several strategies have been designed to combat this problem [ ]. There have been increasing calls by government agencies, such as the US Centers for Disease Control and Prevention, to embrace regular tracking and reporting (audits and feedback) as two of the core elements of stewardship programs [ ].
Audits and feedback  can incorporate behavioral science elements, that is, clinical performance can be reviewed and compared among peers, and this has been widely reported as effective in reducing the inappropriate prescribing of antibiotics [ , , , ]. Despite strong support in the literature, there are at least two important factors that make audits and feedback hard to implement in real-life settings or make them ineffective altogether. First, without proper privacy safeguards, the risk of breaching regulatory compliance with regard to health data confidentiality increases. Additionally, underperforming general practitioners (GPs) may feel exposed and view the exercise as punitive [ ]. The second factor, scalability, is important for efficiently analyzing vast amounts of distributed electronic health record (EHR) data to facilitate the regular scheduling of audit and feedback (A&F) programs. Regular scheduling is an important factor, since evidence shows that if audits and feedback are stopped, their benefits are likely to dissipate [ , ].
This study evaluates a scalable, privacy-preserving A&F system  in clinical settings. The system was deployed in three GP offices and used to generate feedback for 20 GPs on their antibiotic prescribing for selected respiratory tract infections (RTIs), which was viewed in comparison to the average performance of peers. The objective is to evaluate the GPs’ sentiments toward (1) the privacy protections that the system offers, (2) the accuracy and appropriateness of using personal feedback as the basis for comparisons with peers, and (3) self-efficacy and the potential of the intervention to improve the prescribing of antibiotics.
Privacy-Preserving System for Audits and Feedback
The privacy-preserving system contains software components that are deployed at each health institution. The system extracts data daily from the local EHR system and transforms and loads the data into a database that conforms to a common data model .
A third party (denoted as a coordinator) aids the system without learning private information and is trusted to follow protocol specifications. This is a standard security model known as the honest-but-curious adversarial model . In this study, the Norwegian Centre for E-health Research acted as the coordinator. The coordinator accepts queries for the aggregated performance indicators of GPs across health care institutions. The system uses privacy-preserving distributed data mining techniques [ - ] for executing statistical queries on the combined data of health care institutions without allowing any party to view the private data that health care institutions compute locally. Statistical results on the combined data of multiple health care institutions are not considered sensitive information and are therefore stored at the coordinator and retrieved through a web service.
The feedback report is generated locally by combining local personal indicators and the aggregated indicators retrieved from the coordinator. Access to the feedback report is restricted to the respective GP and is provided through a web client or email as an encrypted PDF file, and the decryption key is sent to a mobile phone. We refer interested readers to a study by Yigzaw et al  for an elaborated description of the system.
In 2019, we deployed the system in three GP offices in Norway, and 20 GPs received a single feedback report on their prescribing of antibiotics for selected RTIs. The GPs were then asked to fill out a web-based questionnaire for assessing their perceptions of the feedback received and how the feedback was presented.
Norwegian GPs are responsible for treating a set of patients and refer only those who need more specialized health services to hospitals . Therefore, the 20 GPs had a total of around 19,345 patients on their lists. The number of registered patients varied during the study period because patients sometimes change their GP.
Audits and Feedback
The GPs received a single feedback report on their prescribing of antibiotics for selected RTIs for which antibiotics are generally not recommended if the patient is otherwise healthy (eg, during the first presentation of a current episode of an RTI or when the patient has no significant underlying comorbidity) . Based on the International Classification of Primary Care, Second Edition [ ], the selected RTIs were acute upper respiratory infection (R74), acute sinusitis (R75), acute laryngitis/tracheitis (R77), acute otitis media/myringitis (H71), and unspecified respiratory infections (R83). Acute bronchitis (R78) was also included because it is most often a viral infection, and the use of an antibiotic is rarely recommended.
Feedback was provided for the selected RTI cases between 2015 and 2018; both combined and separate feedback were provided for each of the diagnoses. All statistics were stratified by both year and antimicrobial spectrum (broad- and narrow-spectrum antibiotics). Based on the Anatomical Therapeutic Chemical classification system , narrow-spectrum antibiotics include β-lactamase–sensitive penicillins (J01CE), and broad-spectrum antibiotics include tetracyclines (J01A); β-lactam antibacterials and penicillins (J01C, excluding J01CE); other β-lactam antibacterials (J01D); sulfonamides and trimethoprim (J01E); macrolides, lincosamides, and streptogramins (J01F); and quinolone antibacterials (J01M).
The feedback report contains the number of cases that were diagnosed and treated with antibiotics. A set of performance indicators was presented as a time series graph that compares the performance indicators of a GP with the average indicators of all participating GPs. Performance indicators were based on the indicators proposed by the European Surveillance of Antimicrobial Consumption Network , as follows:
- The percentage of diagnosed cases treated with an antibacterial drug for systemic use
- The percentage of cases treated with narrow-spectrum antibiotics among all diagnosed cases treated with antibiotics
- The percentage of cases treated with broad-spectrum antibiotics among all diagnosed cases treated with antibiotics
A panel of experts with clinical and medical informatics backgrounds developed the questionnaire. As shown in, the questionnaire broadly covers aspects that were derived from the following three theoretical constructs of the Theory of Planned Behavior [ ]: (1) attitude toward privacy and the prescribing of antibiotics, (2) subjective norms (reflection on the accuracy and appropriateness of using feedback as the basis for comparisons with peers), and (3) self-efficacy regarding changing prescribing behavior.
Descriptive statistics were calculated to summarize the results of the categorical questions, including age groups and years of experience.
This study was based on an anonymous questionnaire and therefore did not require ethics review board approval, according to Norwegian regulations. However, all participants gave their informed consent to participate in this study.
The retrospective data that were generated by using the A&F system show that approximately 20% (2924/14,396, 20.3%) of all cases of the selected RTIs were treated with antibiotics, as illustrated in.
A total of 14 GPs responded to the questionnaire, representing a 70% (14/20) response rate. Of these GPs, 36% (5/14) were aged 50 years or older, 57% (8/14) had more than 15 years of experience as a GP, and 79% (11/14) had a specialization in family medicine. The survey results on attitudes toward privacy, audits, and feedback are summarized in.
|Very useful, n (%)||Useful, n (%)||Not useful, n (%)|
|Participant attitudes toward patients’ privacy||10 (71)||4 (28)||0 (0)|
|Participant attitudes toward physicians’ privacy||3 (21)||6 (42)||5 (35)|
|Participant attitudes toward feedback on own prescribing of antibiotics||9 (64)||4 (28)||1 (7)|
|Participant attitudes toward comparison with peers||6 (42)||7 (50)||1 (7)|
We can observe fromthat the GPs were unanimous about privacy protection for their patients, and 64% (9/14) of GPs considered the protection of their own privacy to be very useful or useful.
Most of the GPs (13/14, 92%) reported that feedback on their prescribing of antibiotics and the ability to compare their performance indicators with those of their peers were useful or very useful. The clinicians preferred to receive feedback at regular intervals, such as half-yearly (5/13, 38%) and yearly (7/13, 53%; 1 response was invalid). Most of the clinicians (13/14, 92%) preferred to receive an encrypted feedback report through email rather than having it integrated into the EHR (3/14, 21%), receiving it on a secure website (1/14, 7%), or receiving it on a mobile app (0/14, 0%).
Effects of Audits and Feedback
Of the 14 GPs, 1 (7%) indicated the intention to change their antibiotic prescribing behavior based on the feedback received, 2 (14%) indicated that they did not intend to change, and the rest (n=11, 78%) were unsure of whether they would change.
Most of the GPs (11/14, 78%) wanted their feedback to be stratified by the characteristics of their patient populations and diagnoses. Age and gender were the commonly requested patient characteristics for stratification. The clinicians also requested stratification by chronic conditions, such as chronic obstructive pulmonary disease, asthma, cystic fibrosis, other airway comorbidities, acne, perioral dermatitis, and heart failure.
Perhaps the major highlight of this study relates to privacy preservation, since the A&F system resolves most of the privacy concerns raised in similar studies. Our results show that the GPs value the protection of their own privacy as well as that of their patients, and this finding is in line with those of existing studies . Our results appear to support our initial assumption that protecting the privacy of clinicians may increase their willingness to participate in quality improvement activities.
GPs in Norway often work in small practices with few or no peers; therefore, comparisons with peers across multiple institutions could be especially useful, and the privacy safeguards serve as further incentives for participation.
Our results show that the GPs were in favor of receiving encrypted feedback reports through their emails. This contrasts with the common belief that integrating a clinical decision support system with EHR systems is an important success factor . It is plausible that the GPs assumed that integration with the EHR would trigger distractive feedback alerts more frequently than those triggered by the desired half-yearly or yearly periodic feedback.
Effects of Audits and Feedback
The GPs appeared to value both seeing their antibiotic prescribing statistics and being able to compare these with those of their peers. Although this study did not measure whether the clinicians actually changed their prescribing behavior following feedback, it is likely that most of the participants had a close-to-average prescribing practice and would therefore not have an incentive to change their prescribing behavior. In a larger sample of clinicians, there would be a higher number of individuals that deviate from the mean and have an incentive to change their prescribing behavior.
The GPs requested the stratification of feedback by patient characteristics, such as demographics and chronic diseases, of which both are known to influence the decision to prescribe antibiotics. Stratification provides important information to clinicians on whether their prescribing behavior can be justified based on patient characteristics and enables peer comparisons among clinicians with similar patient populations.
An important limitation is that we could not evaluate GPs’ responses based on their clinical performance, since the questionnaire was anonymous and access to feedback reports was limited to the respective clinician. For example, we were not able to assess how the self-reported intention to change related to an individual’s current clinical performance.
Another limitation is that the feedback reports provided to GPs were not adjusted for comorbidities and age. These two variables are known to influence the decision to prescribe antibiotics, and this may have affected their responses to the survey.
We also noted that the average performance of peers might be skewed if there are outlier clinicians who are far from the mean on 1 side. Therefore, it might be necessary to exclude outliers or provide more statistics, such as SDs and percentiles, to enable detailed comparisons with peers.
The sample size in this study can be considered small but can also be viewed as appropriate for an in-depth case analysis for a pilot implementation of this type.
Recommendations, Implications for Practice, and Impacts
The findings from this study have important implications for practice, especially implications related to quality improvement programs at health care institutions. We single out 3 elements that we recommend for improving systems that support antibiotic stewardship programs. The first and basic element is protecting the privacy of clinicians as well as that of patients. Second, scalability should be considered so as to enable comparisons among peers across multiple health care providers. Comparisons across health care providers are especially useful for small practices like GP offices, since such comparisons require a wide pool of similar patient groups and can be adjusted for comorbidities and age. Finally, human factors should be considered. For example, the frequency of feedback should be considered, since clinicians may not want distractive feedback alerts.
In terms of impact, our system offers a solution to key challenges that hamper antibiotic stewardship programs. It provides privacy guarantees to patients, clinicians, and health care institutions and provides the scalability required to ensure that long-term audits and feedback can be sustainable parts of stewardship programs.
The impact for Norway might be limited, since Norway is a small country with relatively lower antibiotic prescribing rates compared to those of other European countries . For larger countries with differently organized general practices, a higher effect can be expected, since audits and feedback are the most effective in places where the baseline performance is poorer compared to that of the best general practices [ ].
Our results show generally positive sentiments among GPs regarding the potential impact of our system, but it should be noted that this was an initial study with a relatively small number of participants. Therefore, the absolute number of clinicians with very high prescribing rates was likely to be low. In larger samples, we expect that a higher number of physicians would receive an incentive to change their practice.
We are very thankful for the general practitioners who participated in this study. We also would like to thank Peder A. Halvorsen, Torje Henriksen, Andrius Budrionis, and Luis Marco-Ruiz for their contributions to the design of the questionnaire that was used in this study. This work was supported by the European Commission’s H2020 research and innovation program (project number: 826093) and the Research Council of Norway (grant 248150/O70). The funding bodies did not have any role in this study and the writing of the manuscript.
Conflicts of Interest
Questionnaire.DOC File , 170 KB
- Gjelstad S, Høye S, Straand J, Brekke M, Dalen I, Lindbæk M. Improving antibiotic prescribing in acute respiratory tract infections: cluster randomised trial from Norwegian general practice (prescription peer academic detailing (Rx-PAD) study). BMJ 2013 Jul 26;347:f4403 [FREE Full text] [CrossRef] [Medline]
- Gerber JS, Prasad PA, Fiks AG, Localio AR, Grundmeier RW, Bell LM, et al. Effect of an outpatient antimicrobial stewardship intervention on broad-spectrum antibiotic prescribing by primary care pediatricians: a randomized trial. JAMA 2013 Jun 12;309(22):2345-2352. [CrossRef] [Medline]
- Handlingsplan mot antibiotikaresistens i helsetjenesten. Ministry of Health and Care Services. URL: https://www.regjeringen .no/contentassets/915655269bc04a47928fce917e4b25f5/handlingsplan-antibiotikaresistens.pdf [accessed 2022-06-29]
- Robinson JL. Commentary on 'Interventions to improve antibiotic prescribing practices in ambulatory care'. Evid Based Child Health 2006 Jun;1(2):693-694 [FREE Full text] [CrossRef] [Medline]
- McDonagh M, Peterson K, Winthrop K, Cantor A, Holzhammer B, Buckley DI. Improving antibiotic prescribing for uncomplicated acute respiratory tract infections. In: Comparative Effectiveness Reviews, No. 163. Rockville, MD: Agency for Healthcare Research and Quality (US); Jan 2016.
- CDC. Core elements of outpatient antibiotic stewardship | Antibiotic use | CDC. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/antibiotic-use/core-elements/outpatient.html [accessed 2021-06-11]
- Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012 Jun 13(6):CD000259. [CrossRef] [Medline]
- Arnold SR, Straus SE. Interventions to improve antibiotic prescribing practices in ambulatory care. Cochrane Database Syst Rev 2005 Oct 19(4):CD003539 [FREE Full text] [CrossRef] [Medline]
- Meeker D, Linder JA, Fox CR, Friedberg MW, Persell SD, Goldstein NJ, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: A randomized clinical trial. JAMA 2016 Feb 09;315(6):562-570 [FREE Full text] [CrossRef] [Medline]
- Patel SJ, Saiman L, Duchon JM, Evans D, Ferng YH, Larson E. Development of an antimicrobial stewardship intervention using a model of actionable feedback. Interdiscip Perspect Infect Dis 2012;2012:150367 [FREE Full text] [CrossRef] [Medline]
- Linder JA, Meeker D, Fox CR, Friedberg MW, Persell SD, Goldstein NJ, et al. Effects of behavioral interventions on inappropriate antibiotic prescribing in primary care 12 months after stopping interventions. JAMA 2017 Oct 10;318(14):1391-1392 [FREE Full text] [CrossRef] [Medline]
- Gerber JS, Prasad PA, Fiks AG, Localio AR, Bell LM, Keren R, et al. Durability of benefits of an outpatient antimicrobial stewardship intervention after discontinuation of audit and feedback. JAMA 2014 Dec 17;312(23):2569-2570. [CrossRef] [Medline]
- Yigzaw KY, Budrionis A, Marco-Ruiz L, Henriksen TD, Halvorsen PA, Bellika JG. Privacy-preserving architecture for providing feedback to clinicians on their clinical performance. BMC Med Inform Decis Mak 2020 Jun 22;20(1):116 [FREE Full text] [CrossRef] [Medline]
- Bellika JG, Henriksen T, Hurley J, Budrionis A, Marco-Ruiz L, Yigzaw KY, et al. Requirements to the data reuse application programming interface for electronic health record systems. Norwegian Centre for E-health Research. 2017. URL: https://ehealthresearch.no/files/documents/Rapporter/NSE-rapport_2017-10_Requirements-to-the-data-reuse-application-for-ehr.pdf [accessed 2022-06-29]
- Lindell Y, Pinkas B. Secure multiparty computation for privacy-preserving data mining. J Priv Confid 2009;1(1):59-98 [FREE Full text] [CrossRef]
- Hailemichael MA, Yigzaw KY, Bellika JG. Emnet: a system for privacy-preserving statistical computing on distributed health data. 2015 Jun Presented at: SHI 2015, The 13th Scandinavien Conference on Health Informatics; June 15-17, 2015; Tromsø, Norway.
- Yigzaw KY, Bellika JG, Andersen A, Hartvigsen G, Fernandez-Llatas C. Towards privacy-preserving computing on distributed electronic health record data. 2013 Presented at: Middleware '13: 14th International Middleware Conference; December 9-13, 2013; Beijing, China. [CrossRef]
- Yigzaw KY, Michalas A, Bellika JG. Secure and scalable deduplication of horizontally partitioned health data for privacy-preserving distributed statistical computation. BMC Med Inform Decis Mak 2017 Jan 03;17(1):1 [FREE Full text] [CrossRef] [Medline]
- Llanwarne N, Newbould J, Burt J, Campbell JL, Roland M. Wasting the doctor's time? A video-elicitation interview study with patients in primary care. Soc Sci Med 2017 Mar;176:113-122 [FREE Full text] [CrossRef] [Medline]
- Harris AM, Hicks LA, Qaseem A, High Value Care Task Force of the American College of Physicians and for the Centers for Disease Control and Prevention. Appropriate antibiotic use for acute respiratory tract infection in adults: Advice for high-value Care from the American College of Physicians and the Centers for Disease Control and Prevention. Ann Intern Med 2016 Mar 15;164(6):425-434 [FREE Full text] [CrossRef] [Medline]
- PH3C Primary Health Care Classification Consortium. PH3C. URL: http://www.ph3c.org/4daction/w3_CatVisu/en/Articles.html?wCatIDAdmin=8 [accessed 2022-03-10]
- WHOCC - Home. WHO Collaborating Centre for Drug Statistics Methodology. URL: https://www.whocc.no/ [accessed 2022-03-10]
- Adriaenssens N, Coenen S. Disease-specific antibiotic prescribing quality indicators report. European Centre for Disease Prevention and Control. 2010 Sep 10. URL: https://ecdc.europa.eu/sites/portal/files/media/en/healthtopics/antimicrobial-resis tance-and-consumption/antimicrobial-consumption/publications-documents/Documents/ESAC-Net-archive-report_disease_specific_antibiotic_prescribing_quality_indicators.pdf [accessed 2022-06-29]
- Liu C, Liu C, Wang D, Deng Z, Tang Y, Zhang X. Determinants of antibiotic prescribing behaviors of primary care physicians in Hubei of China: a structural equation model based on the theory of planned behavior. Antimicrob Resist Infect Control 2019 Jan 30;8:23 [FREE Full text] [CrossRef] [Medline]
- El Emam K, Mercer J, Moreau K, Grava-Gubins I, Buckeridge D, Jonker E. Physician privacy concerns when disclosing patient data for public health purposes during a pandemic influenza outbreak. BMC Public Health 2011 Jun 09;11:454 [FREE Full text] [CrossRef] [Medline]
- Marcos M, Maldonado JA, Martínez-Salvador B, Boscá D, Robles M. Interoperability of clinical decision-support systems and electronic health records using archetypes: a case study in clinical trial eligibility. J Biomed Inform 2013 Aug;46(4):676-689 [FREE Full text] [CrossRef] [Medline]
- Adriaenssens N, Coenen S, Versporten A, Muller A, Minalu G, Faes C, ESAC Project Group. European Surveillance of Antimicrobial Consumption (ESAC): outpatient antibiotic use in Europe (1997-2009). J Antimicrob Chemother 2011 Dec;66 Suppl 6:vi3-v12. [CrossRef] [Medline]
|A&F: audit and feedback|
|EHR: electronic health record|
|GP: general practitioner|
|RTI: respiratory tract infection|
Edited by A Mavragani; submitted 01.07.21; peer-reviewed by M Salimi, M Davis; comments to author 02.11.21; revised version received 21.03.22; accepted 14.04.22; published 13.07.22Copyright
©Kassaye Yitbarek Yigzaw, Taridzo Chomutare, Rolf Wynn, Gro Karine Rosvold Berntsen, Johan Gustav Bellika. Originally published in JMIR Formative Research (https://formative.jmir.org), 13.07.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.