In the first phase, we used a participatory design approach that was user-centric, visual, iterative, and collaborative, fostering end-user engagement and design thinking [23]. We conducted 2 focus group sessions using the design box method, a technique originally developed for video game design that facilitates end-user collaboration in prototype design and development [24,25]. Given the gaming aspects of VR, we felt that the design box methodology would be best suited for this application. One focus group comprised physical therapists, and the second consisted of individuals who had previously used PT services following back surgery.

The design box method, created by Altizer et al [25] at the University of Utah, advances creative ideation by allowing groups to generate ideas iteratively and design prototype features using a visual tool. It is an inductive design methodology built around the notion that good design solves a problem. This method includes drawing 4 “walls” or constraints on an electronic whiteboard: problem, technology, audience, and aesthetics (Figure 1). The moderator, a member of the research team, gathers input from participants for each wall in sequence, recording their contributions in real time for everyone to see. Once theoretical saturation is reached and no new ideas are generated, design box focus group participants refine the ideas to arrive at a clear design that the entire group supports, otherwise referred to as the “pitch.” By leveraging the 4 walls as constraints, participants are free to pitch any idea that addresses the problem statement, technical needs, audience needs, and agreed-upon aesthetics.

For the physical therapist focus group, we recruited therapists with outpatient experience in providing PT. Using purposive sampling, we targeted physical therapists at the University of Utah in Salt Lake City, Utah, through email and word of mouth. Additionally, we used a snowball sampling technique, asking existing study subjects and local contacts to identify other potential participants. The inclusion criteria were individuals who were aged ≥18 years, English speaking, and licensed to practice PT.

For the second focus group, we recruited individuals who had undergone PT following surgery for back pain. Using both purposive and snowball sampling, we asked health care providers to refer patients who met the criteria. Other inclusion criteria were individuals who were aged ≥18 years, English speaking, and able to meet via Zoom (Zoom Video Communications).

Once the design box focus group sessions were completed, prototype development for the VR application commenced. To facilitate development, the notes and whiteboards from the design box sessions were summarized and analyzed to determine which ideas or features were common and fit within the scope of the project. The analysis of design boxes shares similarities with inductive approaches as the core design of the project emerges from commonalities in and observations of the design box sessions [26]. The core design is broken down into features, which in turn are put into a production backlog, facilitating agile software development [27].

A team of 3 graduate students, 1 producer, and 1 engineer from the Therapeutic Games and Apps Lab at the University of Utah reviewed and aligned the design box results with the capabilities of the Meta Quest VR device (Reality Labs), adding further nuance to the design and backlog. The Meta Quest VR device, a commercial off-the-shelf technology, was selected due to its market prominence, making it easy for translation and adoption. It should be noted that members of the research team were also members of the development team, allowing for negotiation between features that participants in the design box sessions wanted and what was technically feasible and within the project’s scope.

To decide which PT exercise to prototype, the team met with physical therapists to review which exercises were used in postsurgical back pain PT and could be tracked using the controllers and headset of the Meta Quest VR system. The development team at the Therapeutic Games and Apps Lab created a detailed design plan for the leg squat exercise, identified by the PT experts as commonly used in postsurgical back pain PT. The plan was based on 2-week sprint cycles of software development, a key aspect of agile development [27]. Sprint cycles allow for work to be broken down into scheduled iterations, which in turn allow for rapid development and the flexibility to modify any features in between sprint cycles based on user, expert, and team feedback [27]. A total of 5 sprint cycles were executed for this project. Using the development platform Unity (Unity Technologies), the team programmed the prototype’s functionality, integrating VR-specific features such as hand tracking and 6-df movement. Rigorous testing was conducted to ensure smooth performance and a safe, immersive experience, addressing any technical flaws or issues. A functioning prototype for the Meta Quest device was completed and used during the usability evaluation phase.

Following the development of the prototype, participants who had previously undergone PT following surgery for back pain were recruited for a 1-hour in-person session to take part in a usability evaluation. Recruitment strategies included soliciting patients from the University of Utah Orthopedic Center and the researchers’ professional contacts. Using both purposive and snowball sampling, we asked health care providers to refer patients who met the criteria. Inclusion criteria specified individuals aged ≥18 years who were English speaking and had completed PT for postsurgical back pain. Patients currently undergoing treatment for postsurgical back pain were excluded from the study so as to not interfere with treatment regimens.

The usability evaluation comprised several steps, including an interview using a think-aloud protocol and a postuse survey [28,29]. To ensure adherence to study procedures, the research team developed guidance documents for the Meta Quest setup and cleaning between uses, think-aloud protocol questions, postuse survey questions, and safety procedures. Three members of the research team, all trained in usability methods, were present for each of the usability evaluation sessions.

Once eligibility was confirmed and consent was completed, participants were introduced to the general use of the Meta Quest headset and hand controllers. Participants were asked to perform the tasks as instructed by the VR prototype. Subsequently, participants were asked a series of questions and probes in conjunction with the think-aloud protocol to facilitate a deeper exploration of their interaction with the tool. Examples of probes included “What do you think that means?” or “Is that what you expected to see?” The questions were designed to progress from general to more specific information.

Additionally, each participant completed a 10-item validated survey tool, the industry-standard System Usability Scale (SUS), to assess factors such as perceived user performance, perceived decision quality, and perceived task completion time [30]. Responses were collected using REDCap (Research Electronic Data Capture; Vanderbilt University), a secure web application [31-33]. Given the formative and diagnostic nature of this study, our primary focus was on identifying significant usability issues. Following SUS guidelines, we calculated a percentile ranking and average score for the responses.

Conceptually, we used thematic saturation, ceasing recruitment when no new insights or themes emerged from the data [34]. Unlike quantitative studies, qualitative studies typically have smaller sample sizes, and guidance for qualitative sample sizes indicates that above 5 is considered adequate for identifying a large portion of usability issues [35-37]. At the conclusion of the usability evaluation, we advised participants to wait for 30 minutes before attempting to drive home or operate heavy machinery as disorientation or nausea could pose safety risks after using a VR device.

Verbatim transcripts were anonymized and verified by a second investigator. Free-text segments were excerpted from the transcripts for content analysis. Using a securely stored spreadsheet, 3 members of the research team coded the transcripts based on the 9 concepts of the Health Information Technology Usability Evaluation Model [38,39]. To enhance rigor, coders first underwent a group coding session to establish a shared understanding of the coding framework. Instead of calculating intercoder reliability, we conducted iterative consensus coding. Through group consensus, we labeled segments as “positive,” “negative,” and “neutral” for the 9 concepts. Regular meetings with the coders were held to discuss and resolve discrepancies, refine the codebook, and ensure consistent application of the conceptual categories in line with qualitative best practices.

This study received ethics approval from the University of Utah Institutional Review Board (00165806). After the usability evaluation, participants were emailed a US $50 gift card as compensation for taking part. All participants provided written informed consent prior to participation and were informed of the study purpose, procedures, potential risks, and their right to withdraw at any time without penalty. To protect participant confidentiality, all raw data were de-identified prior to analysis. A separate linkage file connecting participants to study IDs was stored in a secure, encrypted location accessible only to the study team. No identifiable information was shared outside the research team.

Seven individuals met the inclusion criteria for the design box focus groups: 3 (43%) physical therapists and 4 (57%) individuals who had undergone PT for postsurgical back pain. Each of the sessions lasted approximately 90 minutes and was audio recorded. The sessions were conducted using Zoom, and the moderator used Miro as the electronic whiteboard. Idea refinement for each of the constraints is shown in Table 1. All participants enthusiastically supported the vision of designing VR-enabled PT exercises.

The pitches from the design box session were reflected in the final prototype design, functionality, and user experience. The VR prototype contained basic menus, buttons, and other interactive elements (Figure 2). The virtual environment resembled a high-technology gym setting with a futuristic avatar used to demonstrate the leg squat exercise (Figure 3). This aesthetic was chosen based on feedback from the design box participants. Participants wanted the environment to be more “fun and exciting” than a standard gym. Additionally, participants mentioned that while some had positive associations with PT gyms, others had negative ones due to postsurgical discomfort.

The VR software leveraged visual and auditory feedback to indicate users’ actions in the software. Buttons would animate to appear pressed, and audio was used to give additional feedback. The users were able to view the avatar (or coach) in front of them demonstrating the exercise and could mirror their movements. Users had the ability to touch buttons or use a virtual laser pointer to click on user interface elements that were out of reach. The VR software enabled users, who may be experiencing physical discomfort, to interact with the user interface without needing to stretch or bend. The experience began with an introduction to the environment and a demonstration of how to perform a leg squat, along with advice on how to do so properly and safely. The users were then able to practice along with the avatar in front of them. Users could decide between 2 modes for their exercise: a timed mode and a counter mode. This choice allowed users to perform as many squats as they wanted to in a given period or take their time and do a number of repetitions without a timer. The system tracked their movements and automatically counted each squat.

Six individuals were recruited for the in-person usability evaluation sessions with 30 to 45 minutes of VR exposure at the orthopedic clinic. Participants were mostly female (n=5, 83%) and aged >50 years (n=4, 67%; mean age 58.3, SD 11.7 y), and 100% (6/6) identified as non-Hispanic White. Overall, participants were enthusiastic about using the VR tool, noting that it was “fun” and “makes exercise less boring.” However, participants reported usability issues related to information needs, especially a lack of clarity on where to move, what to do next, and how many repetitions were completed. Usability issues were also identified concerning the use of the hand controllers and wearing the headset. Representative quotes for each of the 9 Health Information Technology Usability Evaluation Model concepts, along with their descriptions, are shown in Table 2.

The SUS scores were analyzed by averaging the responses from the 6 participants by question (Table 3) and calculating a total adjusted score. Using the Nielsen Norman Group usability guidelines [40], we found a total usability score of 58.3 for the VR prototype, indicating a below-average score [41].

In this study, we identified several challenges while evaluating the usability of a prototype version of the ADAPT XR VR tool. Although some participants gave positive feedback, we found usability issues related to operating the device and navigating the physical space. As the VR environment requires boundaries to enable user movement while maintaining immersion, settings must be applied in advance of use. However, the boundaries in the VR prototype did not match the physical space available for the evaluation sessions. Additionally, the boundaries were overly restrictive, causing participant discomfort and frustration with where to move. Future iterations need to address these boundary issues to ensure accurate usability results as well as participant safety.

Usability was also impacted by participants’ overall familiarity with VR. They found the headset difficult to adjust and uncomfortable to wear. Additionally, there were challenges in using the hand controls and selecting menu options. This was reflected in the SUS scores, which indicated the complexity of the VR tool and the need for significant learning before use. Evaluating the usability of VR tools can be complex given the multiple hardware and software components. However, our usability score of 58.3 is in line with other technology prototype scores, and we believe that it provides an adequate baseline to improve upon [42]. Future evaluations would benefit from offering a general orientation to VR hardware and environments before testing a particular application.

To enhance usability during the development process, we used design box participatory methods to investigate user expectations alongside a usability evaluation of the initial prototype. While the design and development of the exercises were feasible, the usability aspects required more attention than anticipated. As found in other studies, human factors issues, such as poor interfaces, lack of user feedback, and physical limitations, are significant barriers to acceptability in patient-facing technologies and can negatively impact health outcomes [43]. Additionally, ergonomic issues have been cited in VR applications for office work and teaching [44,45]. Canniff and Cliburn [44] reported discomfort with the headset as the most important finding of their evaluation of the Oculus Quest with students. To our knowledge, this is the first study to report similar issues in a health care context. Although we acknowledge that some of the usability issues are limitations of the current VR technology available, in future testing, we intend to place a focus on the physical aspects of the tool to understand the impact of comfort on continued use.

As this study evaluated our initial prototype design, we plan to conduct multiple rounds of prototype testing as we iteratively refine the tool, and we expect usability to improve with further rounds of refinement. We will focus on evaluating a single exercise, the leg squat, within the VR tool to gain detailed insights into its usability by incorporating feedback mechanisms to assess whether the exercise was done correctly and the individual’s ability to balance. This targeted approach will allow us to make specific adjustments and better understand user experiences. Additionally, we will implement a comprehensive onboarding process to familiarize users with VR devices and environments before evaluating the specific application. We believe that onboarding will be critical for VR tool usability until VR devices become more intuitive and are widely adopted. We plan to further investigate how the use of VR tools may exacerbate disparities among different groups, such as technology-savvy individuals, rural communities, and those with a lower socioeconomic status. This will help us identify and mitigate potential barriers to access and usability. We are also interested in investigating how different age groups interact with the VR tool, identifying any unique challenges or preferences that may arise and ensuring that the tool is accessible and effective for users of all ages and abilities.

The findings are constrained by the self-selection of patients interested in new technologies and a reliance on purposive sampling given the need to meet in person, both of which may not have resulted in a sample representative of the broader patient population. Consequently, the usability and acceptability of the technology might be overestimated as individuals with varied levels of technology savviness could encounter more difficulties. Future research should include a more diverse range of participants to ensure that the technology is accessible and effective for all users, particularly those from various care settings, rural areas, and non–English-speaking backgrounds. There may be potential biases as members of the research team also participated in development; however, this was important for the translation of the findings to the design of the prototype. In addition, this work did not explore whether the participants were executing the exercise correctly, which is planned as part of future enhancements. Future stages of the prototype should incorporate methods to evaluate the exercise intervention in the user’s preferred setting (eg, at home), include perspectives from physical therapists, and measure patient outcomes compared to current PT methods.

In this study, we designed and evaluated the usability of a prototype for a VR-based PT intervention. While there was enthusiasm and interest in using a VR tool at home, participants were dissatisfied with the prototype’s usability, particularly related to the device itself. Usability issues were connected to the need for more instructions and difficulties interacting with the VR device. The findings highlight the importance of understanding users’ baseline knowledge of VR and creating effective onboarding materials. Future research should focus on extensive prototype testing and use multiple methods to evaluate the usability and acceptability of VR technology, as well as include an orientation to VR, as it was many participants’ first experience with the technology. Overall, VR tools have the potential to expand care access to rural and underserved communities and increase patient autonomy, making health care more engaging.