Over 4.5 million US adults were diagnosed with liver disease in 2018, and an estimated 44,358 died of chronic liver disease in 2019 [1,2]. Recent estimates indicate that the total annual health care cost of advanced liver disease or cirrhosis is close to US $9.5 billion [3]. Cirrhosis may be accompanied by a host of complications that require a complex, patient-centered treatment plan, including biannual surveillance visits [4,5]. However, patients with cirrhosis are often not diagnosed by their primary care provider or, alternatively, are not referred to specialty care after diagnosis [6]. Owing to the complexity of cirrhosis treatment, a systematic approach to cirrhosis management is critical to preventing further complications and the risk of death.

Health informatics tools have shown promise in the management of complex diseases and can enhance the continuity of care [7-10]. However, it is often challenging to incorporate these tools into existing clinical workflows, with implementation success dependent on ongoing usability assessments [11,12]. In view of the high morbidity and mortality associated with cirrhosis, we conducted a preimplementation formative evaluation to identify facilitators and barriers to implementation of the Population-Based Cirrhosis Identification and Management System (P-CIMS) [13].

P-CIMS, a secure, web-based informatics tool for providers, is designed to ensure efficient cirrhosis case identification in clinical populations, facilitate coordination of care for known cases, and prevent loss to follow-up [13]. Figure 1 presents the overall process workflow chart for P-CIMS. In Figure 1, we describe a scenario in which a provider receives a list of probable cirrhosis cases and must confirm cirrhosis diagnosis before tracking them in P-CIMS.

The purpose of this study is to conduct a preimplementation formative evaluation using qualitative interviews to identify barriers and facilitators of P-CIMS implementation across 3 testing sites, including at the beta testing site.

We conducted a qualitative formative evaluation with key stakeholders at 3 high-volume Department of Veterans Affairs (VA) medical centers in the Sierra Pacific and Southern Central United States (N=10), including a high-volume site at which beta testing occurred. The other 2 sites were chosen based on the criteria for successful P-CIMS implementation. These criteria included the following: (1) established relationships between external facilitators (FK, DLS, and AMM) and key stakeholders; (2) internal facilitators, such as primary care and liver specialty clinical leadership support; and (3) liver clinic patient volume. The sites that were chosen were known to have a higher than average liver clinic volume.

Although the key purpose of this study was to identify potential implementation barriers and facilitators through qualitative interviews, we were also interested in examining usability. As such, we centered on interviews with specific end-users who would be using the tool for clinical care; we followed sample size recommendations from the empirical literature. Guidance by Turner, Lewis, and Nielsen maintains that a minimum of 3 to 5 participants are sufficient for a usability test with the caveat that running additional subjects during the same test is unlikely to reveal new information [15]. Further, other guidance from the quality improvement literature maintains that code saturation can be reached with as few as 9 interviews [16]. Our final sample comprised 10 providers, including 8 physicians and midlevel providers from liver-related specialty clinics and 2 primary care providers who managed patients with cirrhosis.

Content analysis is the primary approach used in the current evaluation [18]. Two analysts (LAM and JW) independently reviewed the transcripts and constructed the codes to describe the data. An iterative consensus process was used to draft a list of subcodes, exemplar quotes, and broader code categories. This list was then used to create a codebook consisting of a priori codes that emerged from the data (eg, suggestions for improving P-CIMS) and code definitions [18]. Individual codes were also deductively linked to CFIR constructs that guided the data collection. For example, suggestions for improving P-CIMS was linked to the CFIR construct Inner Setting: Readiness for Implementation. These constructs served as sensitizing concepts to guide coding, but still allowed for the identification of other salient themes in the data [18].

The codebook was applied for consensus-based coding of the remaining transcripts. Two coders (LAM and JW) met regularly to identify additional codes and refine existing code definitions, with any discrepancies resolved through negotiated consensus. One member of the qualitative team (JW) compiled coded passages into separate transcripts, which were used to derive subthemes. Subthemes were then compiled into matrices for each code, which included exemplar quotes and examples. For example, for the major code suggestions for improving P-CIMS, one subtheme was adding patients to P-CIMS, with descriptive examples, automatic instead of manual entry, and need clear definition of cirrhosis.

Several steps were taken to bolster the validity of data collection and analysis. First, 2 pilot interviews with the target users of P-CIMS were used to refine the initial codebook (See Interview Guide Development and Testing section). This is a step that is often skipped in qualitative research, but it allows investigators to address instrumentation and bias issues before actual data collection occurs [19]. Second, the qualitative team purposively sampled interview participants and interviewed only individuals who would provide the most appropriate and meaningful insights on P-CIMS [20]. To that end, 20% (2/10) primary care providers were interviewed, and specialists (eg, gastroenterologists) comprised most of the sample (8/10, 80% providers). These specialists are the target end-users for P-CIMS, as they work most closely with patients who would benefit from the tool. In contrast, we found that interviews with the 2 primary care providers did not offer unique information in comparison to specialty care providers. Therefore, we chose not to interview primary care providers. Third, the qualitative team employed 2 types of triangulation of analytic findings: theoretical triangulation and investigator triangulation [21]. Theoretical triangulation, defined as the use of substantive theoretical lenses (ie, CFIR) to drive data collection and review research findings, provided a strong basis for the development of both the interview guide and the codebook. Investigator triangulation, or data analysis by more than 2 independent coders with experience in qualitative research (ie, LAM, JW, and DLS) ensured that concurrence for the final list of subthemes and descriptive examples was achieved. Data analyses were iterative and continued until thematic saturation was reached.

The Stanford University Human Research Protection Program reviewed and approved this project as a quality improvement project. All participants provided verbal consent before being interviewed. For this type of study (ie, quality improvement), formal consent is not required.

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The Stanford University Human Research Protection Program reviewed and approved this project as a quality improvement project. The VA Central Institutional Review Board also approved project #14-11 Population-based identification and management of veterans with HCV cirrhosis. For this type of study (ie, quality improvement), formal consent was not required.

All participants gave verbal consent before being interviewed.

Nine interviews took place in person or over the phone and were audio-recorded and transcribed. One interviewee requested that the interview not be recorded, and extensive notes were taken for this interview.

Table 1 presents the main findings mapped to the CFIR constructs of the outer and inner settings and individual characteristics of the individuals. Figure 3 presents the main findings mapped to these CFIR constructs in a descriptive figure.

A preimplementation formative evaluation using the CFIR domains of outer setting, inner setting, and characteristics of individuals identified several key benefits and barriers to implementing a health informatics tool for the management of patients with cirrhosis—P-CIMS—at 3 sites. While participants expressed overall interest in and appreciation for the potential value of P-CIMS, there was concern about the feasibility of implementation. Mainly, barriers in the inner setting—understaffing and workload—posed the biggest challenges in the implementation of P-CIMS.

During the formative evaluation process, providers discussed current strategies for linking patients with cirrhosis to liver clinic specialty care (characteristics of individuals: perceptions of current management of patients with cirrhosis). In the absence of P-CIMS, providers used manual spreadsheets and manual tracking systems to manage scheduled office visits, surveillance testing, and care. For some clinics, the integration of digital tools in addition to regular electronic health record use may be onerous and result in less efficient forms of documentation. For instance, the use of electronic health records to conduct clerical and administrative tasks has been shown to comprise nearly half of some clinicians’ overall workdays [22]. Providers burdened with these tasks may seek out alternatives to P-CIMS to track patients with cirrhosis. Overreliance on nonelectronic health record documentation methods, such as noncoded text notes, has been shown in the past studies of health data [23]. In this evaluation, there were major concerns that adequate implementation of P-CIMS would ultimately slow down clinical productivity (inner setting: readiness for implementation). Participants suggested that having a coordinator may circumvent time barriers but perceived that their departments did not have the resources to hire someone to fulfill this role (outer setting: needs and resources for patients with cirrhosis). However, participants (clinical staff) felt that the tool was useful and that they could use it outside their clinic.

The lack of resources to hire a coordinator was because of the lack of buy-in from leadership in their settings for use of clinical tools such as P-CIMS (Inner Setting: Readiness for Implementation). Providers at 2 locations reported that cirrhosis management was not currently a priority for their leadership, and the importance of cirrhosis was secondary to other competing priorities. Health systems have historically established chronic disease priorities based on the overall prevalence of chronic conditions across patients [24]. The apparent lack of leadership support indicates that health informatics tools such as P-CIMS may be more successfully implemented if they are prioritized and championed in the inner clinic setting, or if their importance for patient outcomes is emphasized to leadership.

A second version of the web-based tracking system, based on feedback from this formative evaluation, is under development. In part, this revised version is under development because preliminary results show that the first version of P-CIMS successfully facilitated linkage to liver clinic specialty care for approximately 30% of patients identified as possibly needing care and who were not already being seen in the clinic [13]. The second version will include key upgrades to P-CIMS, including features that facilitate identification and linkage to care of patients living with cirrhosis, as well as improvements in usability. To facilitate these improvements, data will first be pulled from a national EMR instead of a local VHA EMR. This will facilitate the spread of P-CIMS across VHAs nationally. Second, patient identification can be improved to facilitate linkage and retention in care. The new version will identify patients within 30 days of an inpatient diagnosis of cirrhosis, thus increasing the likelihood that liver clinic providers will follow up and retain these patients in care. This version will also identify patients with no formal diagnosis of cirrhosis in the national EMR data, but those with International Classification of Diseases, Tenth Revision codes for complications of cirrhosis. These more strategic identification methods are another step in the process of linking patients, who might not otherwise be referred, with specialty care. The final change to patient identification will be to include patients with current or past evidence of positive HCV RNA at any point in VHA care (instead of a recent positive HCV RNA) with fibrosis-4 score >3.25. This will enable providers to monitor hepatocellular carcinoma surveillance of patients who have attained virological cure from direct-acting antiviral treatments. Finally, the user interface will be streamlined to decrease the cognitive load of the end user, streamlining multiple tabs into fewer tabs while providing the necessary information to make clinical decisions. Changes to the second version address several limitations of the initial version. Developers and implementers of similar systems should consider these changes, particularly using a streamlined interface for a busy clinic, for future work in the management of chronic diseases such as cirrhosis.

Although we believe this study is a valuable contribution to the literature examining the implementation of web-based health informatics tools, we do want to address some limitations. First, the study sample size was smaller than that recommended by some standards for formative research evaluations [20]; however, we followed best practices outlined for human factors when evaluating informatics tools [15,25,26]. We also reached thematic saturation related to the identification of the key factors impacting P-CIMS implementation [27]. Within these parameters, our sample size was appropriate, given that the purpose of this study was to identify factors that impacted P-CIMS implementation among a purposive sample of providers of cirrhosis and liver care.

A second limitation was that the provider and care team interviewees in the current sample may have been biased in their perception of P-CIMS as a tool. In the preimplementation phase, sites were chosen based on their perceived ability to implement a cirrhosis management tool. Key characteristics of successful site candidates included established relationships with potential liver clinic champions in the inner setting and external factors that facilitated the implementation of such tools. Furthermore, only 20% (2/10) of the primary care providers were included in the sample. Their perspectives may not have been representative of all VHA primary care providers serving patients with cirrhosis. Barriers and facilitators of implementation may differ in sites without these facilitators. Therefore, these findings may have limited generalizability to both other VA settings and non-VA settings, since data were collected from only 3-VA facilities. These results may also not be generalizable to health care settings that do not have the benefit of a national integrated system of data from EMRs.

Finally, we understand that there is a small likelihood that adverse events may occur with the use of P-CIMS, but our findings from this and the larger evaluation do not indicate that the risk of using the tool is higher than the risks associated with current clinical practices used to track and manage patients with cirrhosis. As reflected in the extant literature, many probable cirrhosis cases are not diagnosed with cirrhosis or seen in specialty liver care. P-CIMS streamlined patient identification and created pathways to improve the coordination and continuity of care [13].

P-CIMS is the first informatics tool to leverage EMR data to improve the quality of care for patients with cirrhosis. This tool can be used to inform the design of other clinical informatics tools for a variety of chronic disease conditions that require close tracking and management. However, as is apparent from our qualitative analyses, tools need to be adapted to meet the needs of understaffed clinics or clinics with high workloads. Furthermore, these findings should be used to translate the implementation of health informatics tools for tracking and monitoring chronic diseases in different settings, using tailored implementation strategies as necessary. For instance, future implementation studies could use implementation frameworks such as CFIR, the capability, opportunity, and motivation model, and the behavior change wheel to both: (1) assess major barriers within specific settings, and (2) map these barriers to implementation strategies [28].

In future studies, patients living with chronic diseases such as cirrhosis should be engaged as key stakeholders during the implementation phase of a novel health informatics tool. It is important to elicit feedback on how health informatics tools can either facilitate or hinder chronic disease management. In this study, lack of resources, lack of support, and competing interests were found to be major barriers to the implementation of this informatics tool. Although these barriers may also be found in other settings, it is important to use implementation frameworks to conduct formative evaluations and adapt strategies that would best translate in those settings. The key element of a formative evaluation in this context is that it enables users’ perceptions of the innovation, in the context of their clinical practices, to be incorporated into the design of a successful, sustainable solution to practice problems.