Engaging Stakeholders With Professional or Lived Experience to Improve Firearm Violence Lexicon Development

Introduction

Exposure to firearm violence includes witnessing a shooting, being threatened by a firearm, losing a loved one to gun violence, and sustaining injuries from firearms [1]. Such exposure is associated with adverse physical and behavioral impacts. To better understand health sequelae following firearm violence exposure, some researchers recommend including these exposures in ongoing surveillance and research [2,3]. Although structured data (eg, diagnostic codes) for exposure to firearm violence are largely unavailable, unstructured electronic health record (EHR) data represent a potential source of information collected in the clinical care process [4]. To explore the applicability of such data in surveillance and research, we developed a natural language processing (NLP) pipeline for using unstructured EHR fields, including clinical notes, to identify those with exposure to firearm violence and subsequently understand more about the health impacts of such exposure. A crucial part of formative work on NLP lexicon development is effectively engaging a broad representation of stakeholders to contribute lexicon terms that might not otherwise be known to study teams [5]. In this research letter, we describe the process and outcomes of engaging stakeholders in NLP lexicon development.

Methods

Study Design

We assembled a stakeholder advisory committee (SAC) comprised of patients, advocates, clinicians, researchers, and others with lived and/or professional experience that includes exposure to firearm violence. The SAC structure was designed to include community advocates, patients with lived experience, practicing clinicians, clinical researchers, data scientists, and clinical informaticists. SAC members were recruited via prior collaboration with a study team member, word of mouth, and presentations to patients and clinicians at web-based workgroup meetings. The final SAC composition included 12 members—5 community advocates and/or patients with lived experience, 1 physician, 2 clinician researchers, 2 clinical informaticists, and 2 data scientists.

In April 2024, SAC members reviewed the lexicon of identified keywords that was developed by the study team [4] and informed by MacPhaul et al’s [6] NLP study, which investigated firearm injury intent. Afterward, SAC members suggested additional terms from their lived or professional experience that may indicate exposure to firearm violence. The review of terms and suggestion of new terms were done via an asynchronous, interactive SAC meeting.

New terms identified by the SAC were searched across EHR clinical notes from 7,103,301 patients who were receiving primary or behavioral health care at community-based health centers in the OCHIN multistate network [7]. A maximum of 30 random notes per keyword were reviewed by a study team member to determine whether a note indicated exposure to firearm violence from a powder mechanism [8]. Descriptive statistics were used to summarize results in a table for review, discussion, and adjudication by the study team and SAC.

Ethical Considerations

The Norwich University institutional review board deemed this study exempt. Stakeholders agreed to guide this study and received compensation for participating in monthly meetings; they were not research participants. Stakeholders completed a professional services agreement, which outlined roles and responsibilities, and a nondisclosure agreement prior to joining the SAC. The EHR study data used by the study team for NLP development were a limited dataset. SAC members were not provided access to the limited dataset.

Results

The initial lexicon of identified keywords provided by the study team to the SAC included 19 terms, of which 13 were used in pilot work and 6 were additionally identified by the study team [4]. SAC members identified 35 additional keywords that were not included in the first iteration of the lexicon. Of the 35 terms, 27 had at least one mention in at least one clinical note. Of the 585 clinical notes reviewed, 5 contained 4 SAC-identified terms—bbs, buckshot, firing, and metal pole—and possibly indicated exposure to firearm violence. The study team met to perform the final adjudication for including or excluding each term. Two terms (buckshot and firing) were determined to have sufficient contextual information to indicate true exposure (Table 1). Notes that included these terms were added to the NLP testing and training dataset (Figure 1).

‎

Discussion

Effectively engaging varied stakeholders in NLP lexicon development led to the identification of 2 additional terms (buckshot and firing) that were previously not considered by the study team. Novel clinical notes with these keywords were input into the training and testing dataset to enhance NLP model performance. Although most words suggested by the SAC did not indicate true exposure, the exercise demonstrated that periodic engagement of different stakeholder advisors in artificial intelligence and machine learning research is an important strategy to reduce data bias [9,10].