We conducted a mixed methods evaluation of VA provider and patient attitudes regarding the usability of Vitals and their intent to use. In-person usability testing sessions consisted of a think-aloud procedure and a semistructured interview, followed by a quantitative survey. We collected data in July and August 2023 and conducted analysis in October 2023.

We recruited participants from the primary and specialty care programs of the VA Boston Healthcare System and VA San Diego Healthcare System. A prospective sample of 15 providers (7-8 per site) included 5 primary care and 10 specialty care providers. Based on studies of specialty care telemedicine [19,20,23] and guidance from OCC, 10 specialty care providers were selected from across specialties that could most benefit from Vitals, namely, dermatology, endocrinology (eg, type 1 and type 2 diabetes), oncology, gastroenterology, pulmonology (eg, chronic obstructive pulmonary disease), nephrology (eg, chronic kidney disease), cardiology (eg, congestive heart failure), neurology (eg, functional neurological disorder), anesthesiology, pain management, rehabilitation (eg, physical or speech therapy), and emergency care or urgent care. A prospective sample of 20 patients (10 per site) included 10 “less experienced” patients who had 1-2 VVC visits and 10 “more experienced” patients who had 3+ VVC visits within the past year on tablets (no phones or computers). The sample reflected a variety of health conditions relevant to vital statistics, for example, diabetes or chronic obstructive pulmonary disease, and provided a broad sociodemographic representation. Based on qualitative research standards, “data saturation” is the point at which no new information or themes are observed. We determined that saturation could be reached with 5-10 individual interviews per sample [24,25].

We conducted 30-minute, in-person sessions with providers and 60-minute, in-person sessions with patients. Researchers LG and ER conducted sessions in Boston and researcher DM conducted sessions in San Diego. Sessions consisted of two steps: (1) usability testing through a think-aloud protocol and interview, and (2) a structured survey.

The semistructured interview guides for providers and patients were informed by the UTAUT model, which proposes that performance expectancy, effort expectancy, social influence, and facilitating conditions (positive and negative) influence a person’s intent to use, the strongest predictor of technology adoption [29]. Other themes explored included participants’ attitudes on Vitals’ measurement accuracy, data privacy, system security, and their trust in the VVC app. Interview questions asked whether any training, educational materials, or technical support were needed and probed for barriers and facilitators to Vitals’ acceptance and maintained use (eg, impact on clinical workflow). Table 1 summarizes the constructs, measures, and their sources (refer to Multimedia Appendices 1-4 for interview guides and web-based surveys for providers and patients).

The 26-item survey quantitative survey that concluded each session was composed of 4 validated scales: the Intervention Usability Scale (IUS) [31], the Acceptability of Intervention Measure (AIM) [34], the Feasibility of Intervention Measure (FIM) [34], and the Intervention Appropriateness Measure (IAM) [34]. All measures use Likert-style questions on a 5-point continuum. Based on research on the System Usability Scale [32,33] (the parent scale for IUS), a score above 68 points is considered above average and anything below 68 points is below average [32,35].

For qualitative analysis, audio recordings of usability testing sessions were transcribed. These, along with session notes, were coded thematically using rapid qualitative inquiry method [36,37]. Researchers LG and ER coded interviews, performing inductive and deductive content analysis. A priori categories included UTAUT constructs (performance expectancy, effort expectancy, social influence, and facilitating conditions) and other themes from the literature (see the Measurement and Instrument section). Emergent themes from participants’ think-aloud comments generated additional codes. The research team conducted a thematic analysis that assessed the patterns of attitudes and experiences and reached consensus on interpretation.

For quantitative analysis, we calculated total scores for the survey measures following the published guidelines for scoring. For ease of interpretation, scores for the AIM, FIM, and IAM were converted to a 100-point scale to match the IUS scores. We then calculated descriptive statistics (eg, range and mean) of the total scores on measures. We calculated descriptive statistics of individual items to capture some additional insights into participant responses. Analysis was conducted using R statistical software (version 4.2; R Foundation for Statistical Computing) [38].

All procedures were approved and considered quality improvement by the Institutional Review Board at the VA Boston Health care System in March 2023. The Institutional Review Board of the VA Boston Healthcare System determined this protocol (1685680-1) “VA Video Connect Vitals – Evaluation of Usability and Intent to Use” to be exempt on May 25, 2022. All participants provided verbal informed consent for their participation and audio recording.

We interviewed 20 VA providers whose average age was 47 (range 30-63) years. Over half (12/20, 60%) identified as female. An equal number of providers identified their race as Asian (7/20, 35%) or White (7/20, 35%). Service line representation included primary care (9/20, 45%) and specialty care (5/20, 25%), with remaining indicating a telehealth specific position. Specialty care included urgent care, neurology physical therapy, and surgery. Their experience with telehealth was broadly distributed.

We also interviewed 13 patients with an average age of 67 (range 38-84) years, all of whom identified as male. Race and ethnicity were evenly divided among patients who self-identified as either Black or African American, Hispanic or Latinx, or Asian. All patients had attained at least a high school diploma. Most patients (12/13, 90%) lived 30 to 60 minutes from their VA facility.

As seen in Figure 3, providers and patients highly rated Vitals’ usability (86 points and 82 points, respectively). Overall, providers thought that Vitals was “useful” and “easy to use.”

Providers liked that Vitals “helps to provide some valuable context” and “helps to triage” patient’s general health and particular conditions.

Patients found Vitals both easy to use and more convenient than conventional methods of reading vitals remotely, for example, using a blood pressure cuff or oxygen sensor.

While Vitals usability scored high overall, providers rated 2 items on the IUS slightly lower, indicating the need for technical assistance at the introduction of the Vitals and evidence of Vitals’ accuracy based on its validity (measurement to standard) and reliability (consistently valid measurement over time). During the structured interviews, providers also emphasized that educational materials distributed to both providers and patients before use would be critical to effective use. Examples included a Vitals data sheet, Vitals onscreen prompts on patients’ smart devices to guide them through use experience, and video training on both the VA website and social media platforms, for example, YouTube (Google).

Patients did not express any concerns with usability of Vitals. However, several patients mentioned the challenge of logging in or maintaining an internet connection throughout the visit. Furthermore, some patients mentioned that it would be challenging to hold up their smartphone or tablet for the required 45-60 seconds (to obtain the vitals reading) and suggested that VA provide a tabletop stand to hold and stabilize the device during the reading.

Providers and patients rated Vitals high on the AIM, reflecting their willingness to use Vitals (Figure 3). They perceived Vitals readings as being accurate, resulting in trust. Neither group expressed concern with security. Providers liked the fact that patients could take their own vitals anytime with the app and that it educates and engages them in their health, an important part of patient-centered care.

Patients felt that Vitals was acceptable for their care stating that it helped them keep track of their vitals between visits, was easier than other ways of collecting vitals, and helped with their telemedicine visits. They did not, however, want this to replace the option of seeing a doctor in person. One veteran expressed this qualified acceptance, saying the following:

Patients offered suggestions on how to promote the technology to them, such as having providers explain Vitals’ value in collecting patient metrics, and VA offering materials on how Vitals’ readings compare with familiar consumer monitoring devices they use, for example, Google Fitbit and Apple Watch.

In both the surveys and interviews, providers and patients found Vitals highly feasible and practical for home-based use. One patient expressed that Vitals could also be empowering (Figure 1).

Finally, providers and patients rated Vitals high on appropriateness, that is, fit for purpose, per the IAM (Figure 1). Providers believed that most clinical teams would readily integrate Vitals into their routine workflow because it saves time and delivers accurate, consistently collected vitals, and may reduce reporting errors in the electronic health record.

VA’s Computerized Patient Record System (CPRS) is being replaced by the Cerner Millennium solution that powers the Department of Defense’s Military Health System (MHS GENESIS) [39].

Furthermore, providers from different specialties indicated that Vitals is appropriate in a variety of clinical contexts. For example, several specialty care providers mentioned how Vitals would enhance their workflow, help with billing for video-based visits, and keep patients safer for preparation appointments. Specialty providers said Vitals could be used for video-based physical therapy appointments and medication management with the pharmacy.

Also, primary care providers expressed that Vitals could help improve video-based visits and help streamline the collection of patient vitals. They also expressed interest in an application for follow-up visits.

Overall, health care providers and veteran patients at 2 VA facilities found that contactless remote collection of vital signs presents a usable, acceptable, feasible, and appropriate way to collect patient health information. Our findings highlight some of the similarities and unique perspectives concerning usability and intent to use among health care providers and patients. The inclusion of patients presents a full description of usability of contactless collection of vital signs, which is critical for developing implementation plans. This is one of the first accounts of testing a video-based vitals tool with patients.

First, providers and patients expressed strong support for Vitals’ usability in their discussion and survey scores. Through hands-on experience in a Vitals demonstration, they found the app to be useful and easy to use. Providers liked that Vitals “helps to provide some valuable context” and “helps to triage” patient’s general health and particular conditions. For successful adoption of Vitals, however, providers indicated their need for technical assistance during the Vitals’ introduction. Providers and patients emphasized that educational materials distributed before use would be critical to effective use. Providers would need a detailed report of Vitals’ accuracy based on its validity (measurement to standard) and reliability (consistently valid measurement over time) with references to peer-reviewed articles. Patients would need a clear, concise fact sheet. Providers and patients indicated that they would also benefit from video trainings (at VA provider- and patient-facing websites or YouTube). Providers suggested that patients’ ease of use could be increased if Vitals displayed onscreen prompts on patients’ digital devices (eg, to guide the patient through the Vitals process and display). Patients found Vitals more convenient than conventional remote monitoring because it required less equipment and setup. However, patients cautioned that VA and providers should make clear that they always have the choice of an in-person visit rather than a video-based visit. Providers and patients recognized that use of Vitals could eliminate the cost and inconvenience of some home-based peripherals (eg, blood pressure cuff) and potentially reduce the risk of patients’ misreading their vitals. Patients and providers urged support for veteran patients with mobility challenges (eg, hand tremors and loss of an arm), suggesting that VA offer a platform to steady the patient’s tablet or phone for accurate and consistent readings.

Second, regarding acceptability or willingness to use, providers generally expressed confidence and trust in Vitals’ readings. Still, they requested technical evidence, including citations to academic studies, to assure Vitals’ accuracy. Providers felt that use of Vitals would save time, potentially increase the accuracy of Vitals’ records, and promote more consistent clinical practice in taking vitals at the start of all visits. Patients also found Vitals acceptable and offered suggestions on how to present the technology to make it more acceptable to them, such as training providers to explain the Vitals app and its metrics during its introduction to patients, offering a device stand to steady the cameras of patients’ devices, and encouraging patients to compare Vitals’ readings with those of their Fitbit or Apple Watch.

Finally, providers noted that Vitals’ flexibility makes it feasible, appropriate, and advantageous for implementation in a wide range of clinical contexts, particularly in specialty care. Providers and patients both gave Vitals high marks for its feasibility and were pleased that it could eliminate the use of in-home peripherals and allow patients to take their vitals at any time. Providers believed that most clinical teams would readily integrate Vitals into their routine workflow because it saves time, delivers accurate and consistently collected vitals, and may reduce reporting errors. Providers signaled their readiness to implement Vitals once FDA approval was given and anticipated potential future capabilities. Specialty care providers, in particular, elaborated on how Vitals would enhance their workflow, help with billing for video-based visits, and keep patients safer for preparation appointments. Primary care providers expressed that Vitals could improve video-based visits and standardize vitals collection.

Several limitations to our findings should be considered. The small size of both provider and patient samples may have introduced potential bias. For example, all 13 patient participants were male, and they self-identified as either Black or African American, Hispanic or Latinx, or Asian. Offering some balance in representation, over half of providers identified as female and providers identified across all race and ethnicity categories. Participants were associated with only 2 VA health care systems, both in urban settings that precluded a rural patient or provider perspective. Future work should use larger, more geographically diverse samples of Vitals users, both providers and patients, as well as investigate the experiences of those who tried Vitals once but did not return, to understand why and how to promote ongoing use.

An essential step in VA’s expanded access for veterans is the increase in usefulness and adoption of telemedicine tools such as VVC [11]. The contactless Vitals feature on their smart devices has the potential to make vitals collection faster and easier for VA providers and patients, and provide accurate, real-time vitals reporting and recording. Integration of this feature would first require FDA approval and technical development but offers the potential to support additional clinical use cases, especially for veterans with reduced mobility, those in underserved communities, and those in geographically remote areas to complete appointments when in-person care is not required. This ensures greater access to health care including treatment and prescribed medications and greater health care equity.