Upper limb rehabilitation plays a critical role in restoring functional abilities in individuals with stroke or neurological injuries, enhancing their independence and autonomy [1,2]. Virtual reality (VR) technology has emerged as a promising tool in rehabilitation therapy, offering immersive and interactive features that can improve patient engagement and therapeutic outcomes [3-5]. Through tailored feedback and adjustable difficulty levels, VR-based rehabilitation programs can address diverse patient needs and therapeutic goals [6]. To achieve optimal clinical use, VR-based rehabilitation software must be designed to meet the needs of end users, including health care professionals such as occupational therapists, within their specific work environments. Usability evaluation is a critical step in this process, as it helps identify design flaws, reduce user errors, and enhance operational efficiency and satisfaction [7]. While existing studies on VR-based rehabilitation have largely focused on technical development and clinical efficacy [4], research on usability issues experienced by health care professionals during actual software use remains limited [3] . Specifically, there is a lack of comprehensive analysis regarding the intuitive interaction, learning curve, and practical applicability of such software in real-world clinical settings [7].

Systematic usability evaluation targeting occupational therapists, as key stakeholders in VR-based rehabilitation, is essential to identify and address usability issues. Formative evaluation methods, such as cognitive walkthroughs, provide valuable insights during the early stages of software development, enabling iterative improvements and better user experience [3].

Errors encountered during the use of medical devices are often linked to poorly designed interfaces rather than user mistakes. Although not all user errors result from design flaws, usability evaluations conducted before commercialization can effectively mitigate design-related issues [8,9]. Usability engineering processes play a vital role in identifying and addressing predictable user errors in clinical environments, ensuring the safety and efficiency of medical devices [10,11].

Usability evaluations can be categorized as formative or summative, depending on the timing and purpose of the evaluation [12]. Formative evaluations aim to identify usability issues and guide design improvements during the product development phase, while summative evaluations focus on validating outcomes at the final stages of development through objective data analysis [13]. Formative evaluations commonly use methods such as cognitive walkthroughs, task analysis, usability testing, and heuristic evaluation to explore and address usability challenges in depth [14].

Cognitive walkthroughs, in particular, are highly effective for evaluating software usability. They focus on assessing the system from the perspective of new users, enabling the identification of potential issues such as inconsistent interface behavior or a lack of clarity [15]. By providing insights into users’ cognitive processes, cognitive walkthroughs empower developers to design more user-friendly, efficient, and satisfying interfaces.

This study aims to conduct a formative evaluation of a prototype VR-based upper limb rehabilitation software, focusing on its usability for occupational therapists. By identifying usability challenges and gathering feedback from intended users, this research seeks to provide actionable insights to guide iterative improvements in the software design. Ultimately, this study aims to contribute foundational data for user-centered design and optimize the software for effective clinical implementation.

This study conducted a usability evaluation of a prototype VR-based upper limb rehabilitation software targeted for use by occupational therapists. The evaluation used cognitive walkthroughs and surveys, covering the entire user workflow, including activating the device, setting it up, and completing tasks. The software was assessed as an embedded system, including its hardware components.

The formative evaluation was conducted in the clinical rehabilitation testbed of the Korean National Rehabilitation Center. Each session lasted approximately 90 minutes per participant (Table 1). Before the evaluation, the facilitator introduced the usability testing process, including its objectives, methods, and relevant product information.

Participants were asked about their previous experience with similar devices. Subsequently, they followed a structured script guiding them through a series of tasks using the software. Participants operated the software as instructed, while observers recorded their behaviors and responses throughout the process. Post-testing, participants were asked to evaluate their satisfaction with the system based on the VR-based upper limb rehabilitation software.

The usability testing was conducted in a simulated clinical environment resembling the intended use setting for the device (Figure 1). Environmental conditions, including lighting, temperature, and noise, were measured before the evaluation: lighting at 550 (SD 100) lx, temperature at 24°C (SD 2°C), relative humidity at 60% (SD 10%), and noise level at 50 (SD 5) dBA.

The cognitive walkthrough method evaluates how well users can navigate an interface without previous training or information about the device [16]. In this study, 6 occupational therapists conducted the cognitive walkthrough, assessing the ease of navigation for specific scenarios outlined in Table 2. Observers recorded and analyzed participants’ reactions based on predefined observation sheets. Observation sheets are designed to record problems discovered during each task, user reactions and actions, and system feedback [17,18].

Before the cognitive walkthrough, participants provided demographic information, clinical experience, and previous experience with similar devices. After completing the cognitive walkthrough, participants completed a survey to further assess the usability of the user interface based on the scenarios. The survey consisted of 23 items rated on a 5-point Likert scale (1=very easy to 5=very difficult). Results were analyzed using means and SDs.

The prototype VR-based upper limb rehabilitation software used in this study was designed to improve upper limb function, including shoulder mobility, in patients with stroke or neurological injuries (Figure 2). Patients wear a head-mounted display (HMD; HTC VIVE Pro Full-Kit including VIBE base stations and controllers) and use handheld controllers to perform voluntary movements in a VR environment for rehabilitation (Figure 3). Occupational therapists, as intended users, set up the hardware, operate the software, and assist patients in improving their upper limb functions.

The software includes eight rehabilitation activities (Figure 4): (1) throwing a catching balls, (2) hammering nails, (3) popping bubbles, (4) moving fruits, (5) dodging meteors, (6) throwing balls, (7) playing the xylophone, and (8) stacking blocks.

The minimum computer requirements to run the software are Microsoft Windows 7 SP, Windows 8.1, or Windows 10, with a display resolution of 1280×72 0r higher. Recommended specifications include Microsoft Windows 10 Professional with the same resolution. The hardware used was the VIVE Pro Full-Kit.

The usability evaluation involved 3 usability practitioners (a facilitator and 2 observers), 6 occupational therapists as participants, and 3 mock patients. The facilitator conducted the evaluation, while observers recorded and analyzed participants’ responses. The sample size of 6 participants was determined to balance cost and effectiveness, as this number is sufficient to identify most usability issues [19]. Mock patients were recruited to simulate the role of patients during testing, prioritizing safety by selecting healthy individuals. In the early stages of development, formative evaluations typically involve the use of simulated patients before including real patients. This approach ensures safety and control while being effective in identifying functional issues in devices [20,21]. Furthermore, unlike real patients, simulated patients are trained to provide consistent conditions and responses, thereby maintaining the reliability and consistency of the evaluation process [22,23]. Accordingly, this study employed simulated patients instead of real patients to conduct the evaluation. All participants were informed of the study’s purpose, methods, and procedures before testing and provided signed consent.

Recruitment was conducted through posted announcements at the Korean National Rehabilitation Center. The inclusion criteria encompassed possession of an occupational therapist license and previous experience with rehabilitation medicine devices, such as computerized cognitive rehabilitation programs, whereas the exclusion criteria comprised individuals who did not provide consent to participate in the study. Through eligibility screening, 6 occupational therapists were selected. The mock patients were 3 healthy adults aged 19 years and older and had no experience using the evaluation product. They were provided detailed explanations of the study and coordinated testing schedules. Written informed consent was obtained from all participants.

This study was approved by the Institutional Review Board of the Korean National Rehabilitation Center (NRC-2023-06-041). All participants provided informed consent, and no financial compensation was offered. Data were anonymized to ensure participant privacy and confidentiality.

The clinical experience of the occupational therapists ranged from 6 months to 6 years, with an average of 3 years and 9 months. In addition, they had previous experience using similar medical devices, such as Comcog and RAPAEL (Neofect) (computerized cognitive rehabilitation programs), with their experience ranging from a minimum of 2 months to a maximum of 5 years (Table 3).

A total of 12 usability errors were identified using cognitive walkthrough. In task 4, “Creating Patient Information,” 4 errors were identified. In subtask 4.1, “Locating the Add Patient Icon,” participants were confused as they could not find the icon leading to the screen for adding patient information. In subtask 4.2, “Registering Patient Information,” participants hesitated when they had to enter such details as the diagnosis and dominant hand manually because moving to the next input field required a click on the mouse, as the tab key shortcut did not work. In subtask 4.3, “Guiding Patients’ Consent for Using Personal Information,” participants skipped the data consent step and almost automatically pressed the confirm button without explaining the process of consent for using personal information to the mock patient.

In task 7, “Performing the Activity,” 5 usability issues were identified. In subtask 7.1, “Selecting Activity,” participants attempted to select activities without first selecting the patient, which triggered a popup message indicating “No patient selected.” In subtask 7.2, “Configuring the Activity Settings,” participants committed multiple errors, including failure to estimate the perceived distance between the hand and the object within the virtual reality, and identifying which hand was being used for interaction. Furthermore, although the users clicked on the “User Settings” button, they failed to save the configurations. There were also challenges in adjusting precise decimal values using mouse clicks and dragging, and this time-consuming process delayed the therapeutic session. In subtask 7.4, “Changing Activities,” participants switched activities without saving current data, which led to the loss of unsaved records. In addition, some therapists had difficulty adjusting the settings after completing an activity, as they were unsure whether they needed to press the confirm button before proceeding to the next activity.

In task 11, “Reviewing Statistical Analysis Results,” 3 errors were identified. In subtask 11.1, “Reviewing Data Graphs,” no issues were observed. In subtask 11.2, “Reviewing Activity Data,” participants were unable to review the data because activity records were not saved, and there was inconsistency between the activity names on the playscreen and the statistical analysis screen, leading to further confusion. In subtask 11.3, “Deleting Data” participants clicked on the pie chart and searched for a delete button on the detailed graph screen.

The survey assessing the usability of user interface showed that most participants rated usability as a score of 2 or lower, indicating ease of use on a 5-point Likert scale (1=very easy to 5=very difficult; Table 4).

The following recommendations were suggested from the results of the cognitive walkthrough to enhance usability and user satisfaction with the interface of the VR-based upper limb rehabilitation software. First, enhancements to the graphical user interface (GUI) are essential. A GUI facilitates more intuitive and efficient information presentation using graphics instead of text [24]. This study identified usability issues stemming from the lack of clarity in the “Add Patient” icon, underscoring the need for a more intuitive icon that clearly conveys its function. In addition, improvements in typography, such as optimizing font size and color contrast, are necessary to enhance the readability of activity-setting screens.

Furthermore, to streamline data entry, manual input for all patient information fields should be replaced with input controls such as pickers, which allow users to select from predefined options, or steppers, which enable users to increment or decrement numbers or values using “+” and “-” buttons. In addition, field navigation should be optimized by enabling users to switch between fields using the tab key, thereby facilitating user input. Finally, the functionality for controlling buttons within the software should be visibly displayed on the interface. When developing GUI-based medical software devices, the interface design should aim to integrate clear control mechanisms and simplify tasks, thereby minimizing users’ cognitive load and memory demands [25].

Second, integrating natural user interface (NUI) elements could further enhance usability. NUI refers to an interface that relies on gestures rather than traditional input devices, such as a mouse or keyboard, to control content [26]. While the VR-based upper limb rehabilitation software uses a gesture-based interface using an HMD and controllers, the system does not differentiate between left and right sides. This limitation can hinder therapists from properly administering treatments for patients with unilateral paralysis. To address this issue, the software should include features to either automatically detect the controller’s orientation or provide manual configuration options to distinguish the affected side. Furthermore, therapists reported difficulties in adjusting the perceived distance between the hand and objects displayed on their monitors, highlighting the need for improved distance control mechanisms in the VR environment. Therefore, a VR software interface should be designed to ensure ease of use for both therapists and patients.

Third, the user interface (UI) design requires improvements to enhance the interaction between the medical device and its users. In this study, we observed that therapists used inconsistent methods for adjusting settings and some failed to save their configurations before proceeding with the tasks. Although most medical device users are well-trained and experienced, errors can still occur when they rely heavily on their own previous knowledge. If the system can provide warning alerts when settings are unsaved or incorrect actions are detected, and provide confirmation prompts when actions are correct, unintended errors can be mitigated [27]. Therefore, warning alerts, confirmation popups, detailed instructions, and help guides within the software interface are recommended to prevent usage errors.

Formative evaluation is a type of usability test conducted during the medical device development phase to improve product design. In this study, we conducted a cognitive walkthrough and surveys with occupational therapists as participants to explore potential areas for improvement in the VR-based upper limb rehabilitation software.

Cognitive walkthrough is a usability test method used to systematically identify interactions between users and a system without previous knowledge of the system. It is particularly suitable for evaluating screen-based systems where users must actively navigate the interface [16]. Test tasks should be selected from the user’s perspective, and the sequence of potential actions should first be outlined, followed by a step-by-step analysis of continuous actions to assess whether appropriate actions are taken at the correct moments. Successful interaction is determined by whether the system feedback helps users achieve their intended goals [28].

In this study, we identified usability issues that need to be addressed in the UI of the VR-based upper limb rehabilitation software as a medical device. However, there are some limitations to note. Although both health care providers delivering the treatment and patients receiving it were considered users, actual patients were not included in the participant group. In addition, the therapists involved in the study did not represent a diverse range of clinical experience levels. Despite these limitations, the rehabilitation specialists who participated in this study provided valuable insights, including perspectives from the patients’ point of view. Future research should expand the scope of such work to include a broader range of users and conduct a more comprehensive analysis of the usability across various features and tasks.

VR-based rehabilitation represents a rapidly emerging field in health care. In this study, we conducted a usability evaluation of a prototype VR-based upper limb rehabilitation software. By ensuring a user-centered interface and addressing the identified UI issues and areas for improvement, enhancing the usability of the software as a medical device will contribute to the development of a solution that fully meets the needs of its users. These efforts will not only prioritize user satisfaction and ease of use but also accelerate the commercialization of the VR-based upper limb rehabilitation software. Furthermore, improving user-device interaction will enhance the usability and satisfaction associated with the device, ultimately contributing to the development of a medical device that delivers error-free performance, as intended by the system.