Shared decision-making (SDM) is a valuable and, in some settings, mandated approach to helping patients make informed and value-aligned decisions in medical care. SDM is especially useful in decisions of equipoise when the benefits of intervention do not clearly outweigh the harms and when the value patients assign to these attributes vary [1,2]. Lung cancer screening (LCS) is an evidence-based intervention that meets these criteria [3,4]. However, despite expert guidelines dating from 2014 that recommend LCS for eligible persons [5], LCS uptake and adherence to follow-up protocols have remained low [6]. A number of SDM tools for LCS have been developed [7-10]; however, the implementation of SDM in practice has been limited [11], and the uptake of LCS remains low [12]. A participatory design approach, including patient end users such as US veterans, may increase the uptake of SDM tools and support value-aligned decisions regarding LCS. A user-centered design of SDM interventions guided by usability metrics is required to advance the integration of web-based SDM tools into clinical practice [13].

LCS with low-dose computed tomography has proven effective in decreasing lung cancer mortality in two large randomized controlled trials: the National Lung Screening Trial in the United States [4] and, more recently, the Dutch–Belgian LCS trial [3]. The trials reported a decrease of 20% and 25% in lung cancer mortality among those screened compared with control populations for the United States and Belgian studies, respectively. However, clinical trials also reported harms, including false positive tests, significant incidental findings, and an excess incidence of lung cancer cases indicative of overdiagnosis. False positive tests require follow-up imaging and, in some cases, invasive diagnostic procedures that can cause harm. False positive rates vary across trials and decrease in the later years of screening. The National Lung Screening Trial reported false positive rates of 26.3 %, 27.2 %, and 15.9% at baseline, year 1, and year 2, respectively. The Dutch–Belgian trial reported false positive rates of 19.8 %, 7.1 %, and 9.0% at baseline, year 1, and year 3, respectively [14].

Decision aids (DAs) are structured tools that include a description of decision options, evidence-based benefits and harms associated with each option, a value clarification exercise, and support for a SDM conversation with one’s health care provider [15]. In systematic reviews, DAs have been found to improve the quality of decision-making with respect to the outcomes of knowledge, preparation for decision-making, and value-aligned decisions [16]. However, the implementation of DAs has been limited. A primary barrier has been their integration into the flow of clinical practice [17-19].

Veterans of the United States military are at a higher risk of developing lung cancer than the general population because of older age, higher rates of smoking, and environmental exposures [20]. Veterans are also at higher risk for mental health conditions, including anxiety and posttraumatic stress disorder (PTSD), which increase the burden of LCS, thus affecting the balance of benefits and potential harm from LCS [20]. To address these challenges, we used a user-centered design to develop a veteran-centric LCS decision tool (LCSDecTool) for use in a primary care setting. This tool was designed to be used independently by the patient before the clinic visit (either at home or in the waiting room) with the option of sharing some components with the clinician during the clinic visit. In this study, we report an iterative process of usability testing of a prototype tool and a revised high-fidelity version, with the latter conducted as part of a hybrid type 1 effectiveness-implementation trial. Usability assessment is a key component in the development of web-based decision support tools. Common methods used in usability testing include observation, cognitive interviews, and self-reported feasibility and usability with validated scales [21]. We seek to determine whether veterans find the tool useful to use in the context of a clinical visit and elucidate key aspects of the user experience that had an impact on usability.

We developed the LCSDecTool using the Promoting Action on Research Implementation in Health Services (PARiHS) implementation framework. The PARiHS framework includes the domains of evidence (scientific evidence supporting intervention efficacy), context (the setting in which the intervention is delivered), and facilitation (training and support needed to deliver the intervention) [22,23]. Systematic reviews have established the efficacy of decision aids in improving the process of medical decision-making, as indicated by an increase in knowledge, perceptions of being informed, accuracy of risk perceptions, and clarity about values [24]. We were guided by criteria from the PARiHS framework in the tool design and usability testing for the prototype and high-fidelity LCSDecTool, both of which were evaluated among eligible veterans receiving primary care in Veteran Affairs (VA) Medical Centers and later in the context of a clinical visit. Both focused on the support needed by veterans to access and navigate the tool. The development and usability assessment were also informed by the Technology Acceptance Model (TAM) [25,26]. The TAM posits that people need to perceive the technology as useful and easy to use to continue using it. This model supported our choice of the quantitative outcome measures of the System Usability Scale (SUS) and the End User Computing Satisfaction (EUCS) measure, which include questions in these domains [27-29].

Stakeholder focus groups (clinicians) and structured interviews (veterans) further informed the content and features of the prototype LCSDecTool [30]. The LCSDecTool was designed to (1) use in advance of or during a primary care clinical visit where LCS may be initiated, (2) ensure that veterans understand the key benefits and potential harms of LCS, (3) help veterans weigh the benefits and potential harms of LCS, (4) provide resources for smoking cessation and mental health treatment, (5) support communication with their provider regarding this decision, and (6) include a clinician portal to streamline the use of the tool with a clinician in the clinical setting. The tool was designed for the primary platform of a tablet but has compatibility with a computer or mobile phone. The stakeholder inputs that informed these goals are summarized in Table 1. The data collection form with illustrations from the prototype version of the LCSDecTool is available in the supplemental materials (Multimedia Appendix 1).

We used both quantitative and qualitative methods for usability testing. The SUS and the EUCS scales have been widely used and well-validated to assess interventions in the domains of ease of use and usefulness [28,29,31-36]. The SUS is a 10-item scale developed for the assessment of a broad range of products; scores (ranging from 0=low usability to 100=high usability) correspond to the following adjective descriptors: worst imaginable, awful, poor, okay, good, excellent, and best imaginable [27,37]. The EUCS scale is a 12-item scale that captures the domains of content, accuracy, format, ease of use, and timeliness relevant to a computer application. Scores range from 1=low satisfaction to 5=high satisfaction [28,29]. The Patient Engagement (PE) scale has been used to assess engagement in web-based programs in the medical setting [38,39]. The PE scale assesses engagement in the domains of caring for one’s health, concerns addressed about LCS, understanding LCS guidelines, and understanding information about LCS. Scores on the PE scale range from 1=low engagement to 5=high engagement.

To better interpret the results of the quantitative measures, we used qualitative approaches, including observations of the testing sessions and brief interviews at the completion of the testing sessions. These qualitative approaches have been applied in usability testing to identify barriers to the effective navigation of tools and processing of information [40,41]. The research assistants (RAs) who obtained the qualitative data underwent formal training in qualitative interviewing before the study. This process included 8 hours of training in qualitative methods in the Mixed Methods Research Lab at the University of Pennsylvania for our senior RAs (JP: Bachelors of Science with Major in Biology; JI: Bachelor of Science with a major in Neuroscience). Our junior RA was trained by the senior RAs on the observation of interviews and feedback (JM: Bachelor of Arts with a major in Psychology).

The study was approved by the institutional review board of the participating sites. Participants provided informed consent to complete the baseline survey in phase 1, and the clinicians of participating veterans signed informed consent forms in phase 2 at the time of enrollment in the study. IRB approval obtained from the Michael J Crescenz VA Medical Center, Philadelphia VA (IRB# 01635, IRB #01721, IRB#01780) and the VA Connecticut Healthcare System (MIRB# 02071, MIRB#02240).

The LCSDecTool was designed to create an engaging experience, inform veterans about LCS, support a value-aligned decision, and facilitate communication with their provider about LCS. We found that usability among veterans was good when administered in the context of a primary care clinical visit. These results provide evidence that an older group of US veterans can navigate tablet-based DAs. Our study was noteworthy for engaging veterans in the design of a decision support intervention in a meaningful way. In creating a veteran-centric tool, we involved veterans at every step of development and incorporated their feedback in an iterative process of tool development. Veterans were able to help determine the components of the SDM process and provide reactions and comments that could help other veterans. The motivation to provide feedback to help other veterans reflects a military culture of caring for and deriving satisfaction from helping other veterans [45].

Given that veterans in this age group may not have been familiar with technology, usability testing was particularly important to ensure that they could interact with and understand the information in the tool. Our usability findings were strengthened by conducting the assessment in the setting of a primary care visit where the LCSDecTool was designed to be used. Through this evaluation, we identified areas of strength and areas that require further refinement and modification.

The qualitative feedback obtained from our findings revealed 5 underlying themes. Two of these themes (low baseline awareness and knowledge about LCS that increased after use of the tool and the desire for more detailed descriptions of the LCS process) indicate that usability testing increased awareness of LCS. This is a key step in the process of adopting this evidence-based and provides preliminary evidence that the tool will increase awareness of LCS. Two additional themes (the tool was generally easier to use, but navigation challenges remain and difficulties in understanding some medical terminology remained) indicate the importance of veteran feedback in creating a tool that veterans are able to understand and use. These themes will guide further refinements of the tool. A final theme indicates that the tool may enhance discussions with providers about smoking cessation and mental health. This theme reinforces that veterans perceive a relationship between mental health, smoking cessation, and LCS and further supports the decision to address this relationship in the LCSDecTool.

Our study sample was a particularly vulnerable population in terms of sociodemographic factors (low level of education and income) and health status. Among those in the phase 2 high-fidelity cohort, 47% (20 out of 43) reported a diagnosis of PTSD. Rates of PTSD are known to be higher in veterans than in the general US population. Furthermore, among veterans, the rates of PTSD are higher in those receiving VA care than those who do not [46]. Veterans who receive care in the VA are also known to have lower education and income than veterans receiving care outside of the VA. Given these differences in socioeconomic status and health status, usability testing among a sample of veterans receiving VA medical care focuses our study on a more vulnerable population of veterans.

We observed a decrease in the usability measures from the prototype testing cohort to the high-fidelity testing cohort. There are several potential reasons for these differences. First, in the prototype cohort assessment, quantitative usability measures were collected immediately after completing the tool. In contrast, in the high-fidelity cohort, the user experience and quantitative usability assessment were separated by a clinic visit. The goal of this study was to assess the usefulness of the tool when used in the clinic setting. However, the intervening clinic visit may have decreased the salience of user experience. Of note, the purpose of the primary care visit was not limited to the topic of LCS, although addressing age and risk factors for appropriate cancer screening is an expected component of a primary care visit. Second, the study directed RAs to be more engaged with participants in the earlier prototype testing than in the high-fidelity testing. For example, in the prototype testing, the RAs discussed the participants’ experience with each section of the tool before progressing to the next section. The increased level of RA engagement during the prototype user session may have positively affected user experience.

According to the TAM, users need to perceive the technology as useful and easy to use to continue using it. Our qualitative data provide additional insights on participant perceptions of the tool regarding usability and usefulness. The design of our value assessment measures was based on qualitative studies conducted with veterans regarding how they perceived and valued the potential benefits and harms of LCS [30]. In usability testing, the value assessments were completed without difficulty, with some participants commenting that they enjoyed responding to these scales and found them helpful. Our PE survey indicates that users of the LCSDecTool felt engaged in the decision-making process, a primary goal of any SDM intervention, as in the development of other eHealth applications [35,36]. Our results suggest that users perceived that the tool supported them in caring for their health, addressed their concerns, and informed them about LCS.

In our study, we observed that 100% of the participants completed the use of the LCSDecTool. Completing the tool included moving through all sections, responding to the value assessment questions, and submitting the summary page. In a recent Cochrane review, which was a subgroup analysis of 105 studies involving >31,000 participants, the median effect of a DA on the length of a medical consultation was to lengthen the consultation by 2.6 minutes in comparison with usual care [24]. Although the length of the LCSDecTool did not emerge as a concern among users in our qualitative feedback, the time required to complete the tool could have contributed to lower scores on the SUS and EUCS scales. Our qualitative data suggest that improving navigation and allowing users to choose which sections to review would decrease the time to completion of the LCSDecTool without compromising usability and effectiveness.

Prior studies on cancer screening DAs provide a comparison of the usability scores for the LCSDecTool. Carter-Harris et al [8] developed an LCS DA that included audio and video features and scripts tailored to the user’s smoking status. The tool, named Lung Talk, reported mean SUS scores of 75.7 (SD 7.9), indicating a good level of usability [8]. Coe et al [47] developed a breast cancer screening DA for use among multi-ethnic women. This tool, named Real Risks, reported mean SUS scores of 80.0 (range 50.0-95.0) and 66.3 (range 55.0-75.0) for the English and Spanish versions, respectively [47]. Our study design is unique in reporting usability scores prospectively as the tool moved from the evaluation of a prototype to the evaluation of a high-fidelity version within a clinical setting. In our study, the SUS scores decreased when the tool was integrated into the clinical setting but remained at an acceptable level of usability.

Our LCSDecTool differs from existing LCS DAs in several respects. It was developed using the principles of user-centered design to increase engagement with the tool among a veteran population. This includes the look and feel of the tool, such as using the Department of Veterans Affairs branding, content that acknowledges the mental health conditions that may affect LCS, and smoking cessation referrals specific for veterans receiving care in VA medical centers.

Our study has some limitations. First, data for the phase 2 high-fidelity version were obtained in the context of a pilot hybrid type 1 effectiveness-implementation trial. In this research setting, participants were able to ask an RA for help with the tool if needed. This may not be feasible in routine care outside the context of a research study. Second, the LCSDecTool was tested on a limited number of veterans and clinical sites. Our population reports high rates of mental health conditions and other comorbidities and has particularly low levels of education and income. The findings may not be generalizable to a broader population of veterans or the general US population. Despite these limitations, the strengths of our study include the evaluation of the usability of the LCSDecTool among veterans who are diverse in race, meet eligibility criteria for LCS, and are receiving care in a VA primary care setting.

In conclusion, our study found that a web-based LCS decision support tool developed for and tested among US veterans receiving care in a VA medical center demonstrates an acceptable level of usability. We designed the LCSDecTool for use before and during a clinical visit, incorporating content, formats, and functions that can be used across these settings. The decision regarding LCS requires patients and their providers to consider scientific evidence of benefits and harms, as well as patient values, priorities, and beliefs. Given these complexities, it is important to conduct usability testing of this patient-centered LCSDecTool in its target population. Our findings support the use of this eHealth technology in the primary care clinical setting as a way of engaging veterans, informing them about a new cancer control screening test, and preparing them to participate in an SDM discussion with their provider. Our study further indicates that involving veterans in all phases of the development of the tool extends veteran involvement to the design of SDM processes. Future work is needed to fully address the information and decision support tools that will help veterans understand and apply the principles of SDM for LCS in the context of a comprehensive health care program.