The RCR extracted clinical and sociodemographic data from the EMRs of patients aged 18 years and older who were admitted to one of three neuroscience units at an academic medical center over six months in 2023. These units were selected as their health care teams had received training for serious illness conversations, which encompass communication, presence, needs assessment, and shared decision-making [24]. Exclusion criteria included hospital employees; patients admitted for fewer than three days (as unstructured notes often omit mention of family); those with a history of familial abuse or neglect (due to safety-related restrictions on family engagement); and patients who died during admission, as family engagement at the end of life, while vital for dignity, does not contribute to safe transitions between care settings [12].

Following institutional review board approval and a waiver of consent, a deidentified and encrypted list of eligible patients was extracted from the Research Patient Database Registry, a data warehouse containing EMR data from hospitals within the academic medical system. The database identified 2032 eligible patients. To optimize a diverse sample, the list was stratified by race into three groups [25]. The final convenience sample (N=405) included 135 Black, 135 White, and 135 Asian or Other patients [26]. Of these, 293 patients met inclusion criteria for data abstraction, exceeding the recommended sample size of 192, which calculated using G*Power (version 3.1.9.6; Heinrich Heine Universität Düsseldorf) with 80% power and a medium effect size to detect differences among the stratified racial groups [27].

A literature review of family engagement revealed significant gaps in existing research. It also helped identify and operationalize key variables of interest (ie,family engagement domains; Table 1). Understanding the structure of the medical record guided the identification of proxy measures for each domain. A draft abstraction tool was developed incorporating both structured and unstructured data using the Research Electronic Data Capture (REDCap, version 14.0.27), a secure web-based data management platform [28]. Nurse and physician notes from admission through two days post-admission, as well as two days prior to discharge through the day of discharge, were reviewed, as these are critical transition points for both patients and families [16,29]. To improve reliability and validity, observations and decisions were noted in an operating manual for both phases of the study [20]. Additionally, using intra-rater reliability, an accepted substitute for inter-rater reliability [22], the abstractor performed two passes through the records, followed by a third pass through any abstraction that had less than 95% agreement [17].

Data were analyzed using SAS software (version 9.4; SAS Institute). Structured and unstructured data were reviewed for the presence or absence of family engagement domains, rather than their magnitude. Consult notes (eg, physical therapy, occupational therapy, case management, social work, spiritual care, palliative care, hospice) were reviewed if available, particularly focusing on admission and discharge. Descriptive statistics were calculated for continuous variables and counts and percentages for categorical variables [17]. The χ² tests were used to assess differences between groups.

This RCR required institutional review board approval as it involved the analysis of medical records without direct patient interaction and a waiver of consent was granted (Protocol # 2023P003145). The data was deidentified to ensure privacy. Additionally, a patient with an unusually long length of stay was excluded from the study to protect privacy. Confidentiality was maintained throughout the study by securing data behind institutional firewalls, and access to data was limited to study personnel.

Standard guidelines suggest that the abstraction tool be tested on three to five records to align with the EMR workflow and to address potential reviewer biases and assumptions. During the prepilot, the draft abstraction tool was initially applied to three randomly selected records, where coding errors—such as issues with branching logic—were identified and corrected. Additionally, organizational and formatting changes were made to better align the tool with the EMR structure. The modified tool was then tested on four more records, which identified consult notes that did or did not contain family engagement documentation. Further modifications were made, and the tool remained stable when applied to two additional records. In total, 9 of the aggregate 2032 records were reviewed in the prepilot phase.

For the pilot phase, standard recommendations suggest testing the abstraction tool on 10% of the final sample [28,36,37]. The tool was applied to 39 patient records from the sample population: 13 Black, 13 White, and 13 Asian or Other patients. Six Black and 4 White patients were excluded for being admitted for fewer than 3 days and 6 patients from the Asian or Other group were excluded due to short stays, death during admission, or potential risk of identification due to length of stay. A total of 23 out of 39 patients (59%) were included for pilot phase analysis. There were no significant differences between included and excluded participants when stratified by race. A test of homogeneity indicated that the only distinguishing variable between the groups was the admitting service.

Using the refined abstraction tool, the RCR identifies which domains of family engagement are documented in the EMR, who records the information, and where it is located [17]. Development of the tool occurs in two distinct but complementary phases—prepiloting and piloting—each playing a critical role in refining the data abstraction process and enhancing the tool’s accuracy and usability.

Prepilot testing involved an initial evaluation to identify issues such as unclear instructions or data elements, with a small group providing feedback for adjustments [22]. Following this, pilot testing with a larger, more diverse sample assessed the tool’s consistency, accuracy, and real-world feasibility, ensuring its readiness for full-scale data collection. These two distinct phases were essential for ensuring the abstraction tool’s reliability and effectiveness in extracting relevant data.

Key strengths of RCR methodology include its flexibility, cost-effectiveness, ability to access unique information, reduction of recall bias, and potential for creating randomizable sample frames [19]. Beyond epidemiological and clinical research, RCR is useful for evaluating care patterns and informing training [19,20,28]. However, a notable limitation was that for an RCR, data are obtained twice from the patient, family, or clinical scenarios; first, the clinician gathers the data and second, they document the data. Finally, while every effort was made to optimize sample diversity, stratifying by race—a social construct—can introduce bias if patients did not self-identify their race. These characteristics should be kept in mind during the prepilot and pilot phases of the abstraction tool development.

The prepilot and pilot phases were essential for refining the abstraction tool used in this RCR. These phases allowed the tool to be optimized for efficiency and clarity, addressing potential problems, and improving data collection. The prepilot phase involved an initial evaluation to identify issues such as unclear instructions or missing data elements, with a small group providing feedback for adjustments. On the contrary, the pilot phase involved a more extensive and structured evaluation, with a larger sample size and more diverse set of users. The final abstraction tool integrated both structured and unstructured data, capturing family engagement across multiple disciplines, including nursing, case management, physicians, and ancillary consultants. The focus remained on critical transition points in care (ie, admission and discharge). The two distinct phases ensured that the tool was clear, efficient, and comprehensive, setting the stage for the larger study.

This study addresses a previously unexamined aspect of RCR. Describing the distinction between the prepilot and pilot phases increases transparency and increases the rigor and reproducibility of the methodology. The description of the iterative testing within each of the two phases, indicating that it was only adopted after careful systematic evaluation, also highlighted the learnings and rationale underlying specific design decisions. This step-wise approach to RCR helps to build best practice standards and inform policy while documenting family engagement.